Concept networks and systems and methods for the creation, update and use of same to select images, including the selection of images corresponding to destinations in artificial intelligence systems

ABSTRACT

Systems and methods for concept based searching or recommendation are disclosed. More particularly, embodiments of a concept based approach to the search and analysis of data, including the creation, update or use of concept networks in searching and analyzing data are disclosed, including embodiments of the usage of such concept networks in artificial intelligence systems that are capable of utilizing concepts expressed by users to return or evaluate associated images.

RELATED APPLICATION(S)

This application claims a benefit of priority under 35 U.S.C. § 119(e)from U.S. Provisional Patent Application No. 62/568,604 filed on Oct. 5,2017 by inventors Pan and Tosik and entitled “Concept Networks andSystems and Methods For the Creation, Update and Use of Same to SelectImages, Including the Selection of Images Corresponding to Destinationsin Artificial Intelligence Systems”. The entire contents of theabove-referenced application are hereby fully incorporated by referenceherein for all purposes.

TECHNICAL FIELD

This disclosure relates generally to systems and methods for conceptbased searching. More particularly, this disclosure relates toembodiments of a concept based approach to the search and analysis ofdata, including the creation, update and use of concept networks insearching and analyzing data. Even more specifically, this disclosurerelates to embodiments of systems and methods for the creation, updateand use of concept networks in searching to increase the accuracy orrelevance of search results. Additionally, this disclosure relates tothe use of such concept networks in artificial intelligence systems,including travel or real-estate related artificial intelligence systemsthat are capable of utilizing concepts expressed by users to returnassociated destinations or corresponding images.

BACKGROUND

Despite the amount of information available in today's Internetenvironment, a number of problems exist. One of the most significantproblems is how to locate and correlate data of interest that may beavailable at a single web site or distributed across a number ofdisparate web sites. In fact, in many cases the greater the amount ofinformation available, the greater the difficulty in locating andcorrelating such information. These problems are exacerbated because, inmany instances, web sites pertinent to a given context may be siloedwith respect to their functionality or information and offer limitedstatic searching interfaces. Thus, while it may be desirable to a userof such web sites to correlate information available at one web sitewith information available at another web site there is no way toaccomplish such correlation that does not involve the manual location ofdata of interest and the tracking and correlation of such data acrossweb sites by the user himself.

Within certain contexts such as travel or real-estate these problems maybe particular prevalent and persistent, in no small part because of thehistorical development and current state of the Internet environmentwith respect to these contexts. As but one example, the travel industryhas grown significantly over the past two decades. Despite the rise ofthe Internet and Internet based commerce over that same time period,travel service providers (e.g., airlines, hotels, rental car companies,tour providers, etc.) were relatively slow to migrate their offeringsonline. Though the travel service providers did eventually establishonline presences, almost universally these travel service provider websites function in the same manner. The web site provides an interface bywhich a traveler may provide a set of static itinerary data such asdates, point of origin, destination and number of travelers. The travelservice provider web site then provides a listing of servicescorresponding to the traveler provided static data, such as a listing ofavailable flights or hotel rooms.

With the proliferation of travel service providers, however, came anumber of problems. Foremost among these problems was the amount of datathat was now available to travelers. Specifically, travelers now couldobtain travel service offerings through a large number of atomic websites. So, for example, to obtain information on flights to adestination, a traveler would visit the web sites for each airline andobtain travel offerings (if any) for their particular itinerary. Asthere are literally thousands of airlines operating worldwide, it can beimagined what a herculean task it is to not only obtain this travelservices data from each individual web site but, additionally, tocompare the various travel service data amongst the various airlines.The same problems hold true almost no matter the type of travel servicebeing provided.

To attempt to mitigate some of the issues with the myriad number oftravel services, certain intermediary or aggregator web sites (e.g.,Expedia, Hotwire, Travelocity, Kayak, etc.) have arisen. Essentiallythese sites functions as search aggregators, allowing travelers tosearch multiple independent travel service providers from a single site.The basic functionality of these aggregator sites is, however, stillunchanged from the brute force methods of yore; the user must specify aset of static itinerary data through the interface of the searchaggregator. The intermediary aggregator site, in turn, searches the dataof each of the independent travel service providers for correspondingtravel services associated with the traveler's static itinerary data andreturns these results to the traveler through the intermediary site.

These intermediary aggregator sites have thus not alleviated theproblems inherent in the implementation of travel service related websites. This is the case at least because it cannot be sure that all thetravel service related web sites are being searched by theseintermediary sites, and because these intermediary sites are usuallyspecific to one aspect of travel service (e.g., flights, hotels, etc.),or are only capable of searching one aspect at a time. Travelers musttherefore still deal with the prospect of searching and vetting a largeamount of travel service data from a variety of travel related websites.

More importantly, however, a bigger problem remains: all of these travelrelated sites function in the same basic manner. A traveler must enterstatic itinerary data and the travel related web site returns travelservices specific to that static data. These travel services relatedsites provide no insight to the user; they only return travel serviceoptions related to what the user already knows (e.g., theirdestination).

This situation compounds the already difficult problem of travelplanning. The traveler must first determine such things as theirintended destination before making these plans. Such determinations arenot straightforward: often time the traveler has an idea or thoughtsabout what type of travel he would like to make, without any concretedestination in mind. The traveler must then engage in preliminaryresearch even to determine their itinerary before engaging with thetravel service related web sites. Given the number and variety ofgeneral travel related web sites, this is often a daunting and timeconsuming task.

While the travel related context serves as a particularly germaneexample of the problems of knowledge location or amalgamation, it willbe understood that similar problems manifest in multiple contexts in theInternet world. As yet another example, in the real-estate relatedcontext it may be difficult to glean insight into real-estate or thereal-estate market given the limited functionality of real-estaterelated web sites. Moreover, as real-estate is not a fungible good orservice, the ability to obtain or correlate data on real-estate that isspecific and pertinent to a particular user may be especially difficult.

What is desired, therefore, are systems that may dynamically providepersonalized and insightful knowledge, including such systems that maybe usefully applied in particular contexts, including for example, atravel related or real-estate related context.

SUMMARY

Attention is thus directed to the artificial intelligence systemsdisclosed herein. These artificial intelligence systems may utilize achat driven interface that includes a chat portion and a displayportion. The user may interact with the chat portion of the interface ina human understandable and comfortable manner (e.g., through naturallanguage). The artificial intelligence system uses the context of theinteractions in the chat potion of the interface to drive thepresentation of content in the display portion of the interface suchthat the content presented in the display portion of the interfacereflects the intents or concepts expressed in the chat portion of theinterface (e.g., in the last interaction, or historically in previousinteractions).

Specifically, according to embodiments, a user's current interactionwith the chat portion of the display may be evaluated in associationwith the context of historical interactions to determine userpreferences, including intents or concepts expressed by the user duringthe current, or previous, chat sessions. Using these user preferences,relevant content can be determined to present to the user through thedisplay portion of the interface along with a relevant responseinteraction to present through the chat portion of the interface. Thechat portion of the interface and the display portion of the interfaceare thus kept in synchronicity.

Additionally, and importantly, the content determined for presentationto the user in the display area is based on the user's intents orconcepts expressed in a natural, human understandable manner (even thosewhich may not be directly expressed by the user), not (solely) on staticdata provided by the user. Accordingly, the content presented to theuser in the display portion of the interface may include contentproviding insights, alternatives, supplemental information or additionaldata of which the user was previously unaware at the outset of theirinteraction. The content displayed can be also be refined in an ongoingmanner based on the concepts or intents expressed in a humanunderstandable and comfortable manner by the user in the chat portion ofthe display. This type of interaction may enable a level of trust withusers. In this manner certain tasks, such as those involved in travelplanning or the like, may be reduced and even, in some instance, madeenjoyable.

Moreover, using embodiments of these artificial intelligence systems achat driven interface that includes a chat portion and a display portionmay be distributed through a variety of channels. In particular, thechat driven interface may be presented in a display container (or unit)on a third party web page or proprietary web site or applicationalongside content provided by the third party web site. The displaycontainer may be of the size typically used to display a web basedadvertisement such as a 600×300 size container (often referred to as anad container or ad unit). The chat driven artificial intelligence systemmay utilize the context of the third party web site, including forexample, the entities (including concepts) and destinations expressed inor derived from the content of the third party web page or site toinitialize and drive the chat driven interface.

In certain embodiments, the context of the web page or site on which thechat driven interface is presented may be used to select content from anadvertiser to present in the display portion of the chat driveninterface presented through such a container based chat driveninterface. The advertiser content selected may be associated with thecontext of the third party web page or site and, additionally, may berefined through the user's natural language interaction with the chatportion of the chat driven interface. Thus, the initial state of thechat driven interface (based on the context of the third party web pageor site in which it is presented) serves to attract the user's attentioninitially as it is related to the context of the content they beactively seeking on the third party web page. The artificialintelligence system may subsequently use the context of the interactionsin the chat potion of the interface to drive the presentation of contentin the display portion of the interface such that the content presentedin the display portion of the interface reflects the concepts ordestinations expressed by the user in the chat portion of the interface.To further enhance the suitability of such content, the context may alsobe used in association with other data such as the user's historicaldata (e.g., past interactions with the artificial intelligence system)or other aspects of the user's online presence.

Embodiments of the chat driven interface as presented may thus improvethe efficiency of using a computing device by bringing togetherdisparate data that may typically be accessed independently accessed (ifat all) in disparate location into a chat driven interface which can beaccessed directly in a single interface, whereby displaying selecteddata of interest in the display portion of the interface allows the userto see the most relevant data without having to navigate between sites,windows or separate interfaces. Accordingly, the speed of a user'snavigation through various views and windows can be improved becauseembodiments of the chat driven interfaces as presented herein saves theuser from navigating to the required sites, opening them up, and thennavigating within those sites to enable the data of interest to be seen.Rather than paging through multiple screens or navigating aroundmultiple web sites and manually correlating data of interest betweenthese web sites, interactions with only a single chat driven interfacemay be needed to access desired data.

As one goal of such artificial intelligence systems may be to select andprovide content reflective of the concepts or intents (including, forexample, destinations) expressed by the user it may therefore bedesirable in many instances to premise the search for items and relatedcontent on a knowledge base including concepts (referred to herein as aconcept network). This is the case at least because the concepts thatmay be used for searching for items and related content are notnecessarily expressly provided by the user. Instead, in many cases theconcepts used for this searching may be derived from the naturallanguage statements of the user (e.g., as provided in a chat interface)or other sources. Accordingly, the search for items for which contentmay be presented is based on those derived concepts, and is not atypical keyword driven search. Moreover, as the searching is based onthese concepts, it may be desirable to not only determine itemsassociated with concepts but, additionally, to determine a degree ofaffinity that these items have with those concepts, which also cannot beascertained through a keyword based search.

To those ends, among others, attention is now directed to embodiments ofconcept networks (or nets) disclosed herein, and the creation, updateand use of these concept networks, including their use in travel orreal-estate related (or other types of) artificial intelligence systemsto select and display one or more destinations. Embodiments of theseconcept networks may include a graph comprised of nodes and edges, wherethe nodes may include concept nodes and the edges may includerelationships between these concepts nodes and other nodes, includingother concept nodes or nodes associated with various entities. In atravel related context, for example, these nodes may represent itemssuch as hotels, destinations, flights, attractions, activities, etc. Therelationships indicated by the edges of the concept network may beassigned weights or scores indicating a degree of affinity between thenodes related by a relationship, including, for example, the degree ofaffinity between a concept represented by a concept node and an itemnode to which that concept node is related. Additionally, therelationships may be directional, such that they may only be traversedin a single direction, or have different weightings depending on thedirection in which the relationship is traversed or the nodes related.

One particular embodiment of a concept network may be referred to as atrie. A trie can be thought of as a search tree data structure used tostore a dynamic set or associative array of concepts. Specifically, incertain embodiments, the trie may be a concept network including ahierarchical set of concept nodes arrayed as a tree structure. Thus,each node at each level of the tree may be a concept node representing aconcept that is a sub-concept of a parent concept node (as representedby the presence of an edge denoting that the node is a sub-node of theparent concept node) and where each node may have one or more sub-nodeseach representing a sub-concept of that node. In this manner, a trie canbe traversed from any node to any other node. Weights or scores (usedinterchangeably herein) can be generated between any two concepts in atrie. In one embodiment, such a weight may be based on the number ofedges (or distance) between two concepts, the number of levels of thetrie that must be traversed, the direction of traversal (e.g., either upthe trie or down the trie) to navigate between the two nodesrepresenting those concepts or other traversal or traversal distancebased criteria.

In one embodiment, a concept network may be created or updated in anydesired manner, including using a set of rules or a machine learningalgorithm. These rules can take into account queries for particularconcepts, a frequency of appearance of concepts or items, the number oftimes particular paths through the concept network are traversed whensearching, or user interactions with content (e.g., associated withcertain items or concepts) that is presented to the user. These rulescan serve to add new concept nodes to the concept network, establish newedges (relationships) or weights between concept nodes and other nodes,or adjust the weights on existing relationships.

Embodiments of such a concept network can thus be searched to determineitems (and related content such as images) associated with one or moreconcepts. Moreover, the degree of affinity between the concepts anditems may be determined using the weights of the relationships in theconcept network. In particular, concept networks including nodesrepresenting travel related items such as hotels or destinations may beused in travel related systems to search for and provide these travelrelated items (and related content) associated with one or moreconcepts. In this manner, the travel related content related to theconcepts expressed by, or determined from, a user may be utilized toprovide insightful items and content to the user.

Specifically, in certain embodiments, searching interfaces to a conceptnetwork may be provided whereby one or more concepts may be provided.These searching interfaces may construct a search for a particular typeof item (e.g., a destination or hotel) based on the concepts receivedand return an ordered list of items (and related content) based on thedegree of affinity between the received concepts and the items asrepresented in the concept network. The determination of a degree ofaffinity between a concept and an item can be based on both the numberof “hops” (i.e., distance) or intervening nodes along the path traversedbetween the node representing the concept and the node representing anitem and the weights of each relationship in the path.

Utilizing embodiments of a concept network, such searches may accuratelyreturn real-world travel related items and related content associatedwith concepts to a user. Moreover, these concept networks may beautomatically updated without user involvement based on a wide varietyof criteria. Additionally, embodiments may employ certain types ofrelationships, types of concept networks, storage of the conceptnetwork, or divisions of the concept network, to achieve greater speedor reduce the processing power required or memory usage required in thestorage, searching or other processing accomplished in association withsuch creating, updating, searching or storing concept networks.

In certain embodiments, embodiments use an API microservice capable ofreceiving concepts and returning destinations or receiving a destinationand responding with concepts.

These, and other, aspects of the disclosure will be better appreciatedand understood when considered in conjunction with the followingdescription and the accompanying drawings. It should be understood,however, that the following description, while indicating variousembodiments of the disclosure and numerous specific details thereof, isgiven by way of illustration and not of limitation. Many substitutions,modifications, additions, or rearrangements may be made within the scopeof the disclosure without departing from the spirit thereof, and thedisclosure includes all such substitutions, modifications, additions, orrearrangements.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings accompanying and forming part of this specification areincluded to depict certain aspects of the disclosure. It should be notedthat the features illustrated in the drawings are not necessarily drawnto scale. A more complete understanding of the disclosure and theadvantages thereof may be acquired by referring to the followingdescription, taken in conjunction with the accompanying drawings inwhich like reference numbers indicate like features and wherein:

FIGS. 1A-1J are depictions of embodiments of an interface for use withembodiments of a chat driven artificial intelligence service used in atravel related context.

FIG. 2 is a block diagram of one embodiment of a computer networktopology that includes an embodiment of a chat driven artificialintelligence system.

FIG. 3 is a block diagram of one embodiment of an orchestrator of a chatdriven artificial intelligence system.

FIG. 4 is a block diagram of one embodiment of a computer networktopology that includes an embodiment of a chat driven artificialintelligence system.

FIG. 5 is a block diagram of one embodiment of a computer networktopology that includes an embodiment of a chat driven artificialintelligence system.

FIGS. 6A, 6B and 6C are depictions of a concept network.

FIG. 7 is a block diagram of one embodiment of a chat driven artificialintelligence system.

FIG. 8 is a flow diagram illustrating one embodiment of a method forsearching for destinations.

FIG. 9 is a flow diagram illustrating one embodiment of a method forrecommending destinations.

FIGS. 10A, 10B and 10C are an exemplary screenshot of an Elasticsearchquery for the region of Cape Town.

FIGS. 11A and 11B are a screenshot showing an overview of an exampleDynamoDB database table.

FIGS. 12A and 12B are a screenshot showing details of an exampleDynamoDB database.

FIG. 13 is a flowchart depicting one example of a process for selectingand ranking images.

FIG. 14 is a flowchart depicting an embodiment of a method for rankingimages.

FIG. 15 depicts an example of an image that has been analyzed toidentify predicted concepts and probabilities.

FIG. 16 depicts a portion of an example trie concept network.

DETAILED DESCRIPTION

Embodiments and the various features and advantageous details thereofare explained more fully with reference to the non-limiting embodimentsthat are illustrated in the accompanying drawings and detailed in thefollowing description. Descriptions of well-known starting materials,processing techniques, components and equipment are omitted so as not tounnecessarily obscure the embodiments in detail. It should beunderstood, however, that the detailed description and the specificexamples are given by way of illustration only and not by way oflimitation. Various substitutions, modifications, additions and/orrearrangements within the spirit or scope of the underlying inventiveconcept will become apparent to those skilled in the art from thisdisclosure.

Before discussing embodiments in more detail a brief description of thecontext in which embodiments can be utilized may be helpful. Asmentioned above, travelers must usually determine such things as theirintended destination or lodging before making travel plans. As mentionedabove, a significant problem in today's Internet environment is how tolocate and correlate data of interest that may be available at a singleweb site or distributed across a number of disparate web sites. Theseproblems may be particularly pernicious as web sites pertinent to agiven context may be siloed with respect to their functionality orinformation, and offer limited static searching interfaces. As anexample of but one manifestation of such problems, in the travel relatedcontext travelers must usually determine such things as their intendeddestination or lodging before making travel plans. Such determinationsare not straightforward: often time the traveler has an idea or thoughtsabout what type of travel he would like to make, without any concretedestination or lodgings in mind. The traveler must then engage inpreliminary research even to determine their itinerary before engagingwith the travel service related web sites. Similarly, in the context ofreal-estate, a tenant or purchaser may need to engage in a large amountof research regarding lodgings for rent or sale, prices, geographicalareas, schools, or other pertinent topics before engaging with realestate web sites. Given the number and variety of general travel relatedor real-estate related web sites these are often daunting and timeconsuming tasks.

What is desired, therefore, are systems that may dynamically providepersonalized and insightful knowledge, including such systems that maybe usefully applied in a variety of contexts, including, for example,travel or real-estate related contexts.

To those ends, among others, attention is now directed to the artificialintelligence systems disclosed herein. These artificial intelligencesystems may utilize a chat driven interface that includes a chat portionand a display portion. The user may interact with the chat portion ofthe interface in a human understandable and comfortable manner (e.g.,through natural language). The artificial intelligence system uses thecontext of the interactions in the chat potion of the interface to drivethe presentation of content in the display portion of the interface suchthat the content presented in the display portion of the interfacereflects the intents or concepts expressed in the chat portion of theinterface (e.g., in the last interaction, or historically in previousinteractions).

Specifically, according to embodiments, a user's current interactionwith the chat portion of the display may be evaluated in associationwith the context of historical interactions to determine userpreferences, including intents or concepts expressed by the user duringthe current, or previous, chat sessions. Using these user preferences,relevant content can be determined to present to the user through thedisplay portion of the interface along with a relevant responseinteraction to present through the chat portion of the interface. Thechat portion of the interface and the display portion of the interfaceare thus kept in synchronicity.

Additionally, and importantly, the content determined for presentationto the user in the display area of the interface is based on the user'sintents or concepts expressed in a natural, human understandable manner(even those which may not be directly expressed by the user), not(solely) on static data provided by the user. Accordingly, the contentpresented to the user in the display portion of the interface mayinclude content providing insights, alternatives, supplementalinformation or additional data of which the user was previously unawareat the outset of their interaction. The content displayed can also berefined in an ongoing manner based on the concepts or intents expressedin a human understandable and comfortable manner by the user in the chatportion of the display.

Before delving into more details regarding the implementations ofspecific embodiments, it will be helpful here to now discuss embodimentsof such artificial intelligence system that may be usefully applied inthe context of travel. Embodiments of such systems may also beunderstood with reference to U.S. patent application Ser. No. 16/015,104filed Jun. 21, 2018, entitled “Interaction Driven ArtificialIntelligence System and Uses for Same, Including Travel for Real EstateRelated Contexts” and U.S. patent application Ser. No. 16/015,112 filedJun. 21, 2018 and entitled “Interaction Driven Artificial IntelligenceSystem and Uses for Same, Including Presentation Through Portions of aWeb Pages” which are hereby incorporated by reference herein in theirentirety for all purposes.

Referring first then to FIGS. 1A-1D, embodiments of an interface for atravel related chat driven artificial intelligence system are depicted.For example in FIG. 1A, interface 100 may include a chat portion or area110 and a display area 112. The user interacts with the chat drivenartificial intelligence system through the chat area 110. In particular,a human understandable natural language conversation may occur in thechat area 110 such that the user may iteratively provide an interactionand the chat driven artificial intelligence system may provide aninteraction prompt or a response interaction. These interactions can beexchanged between the user and the chat driven artificial intelligencesystem in the chat area 110.

The interactions taking place in the chat area 110 drive the display ofcontent in display area 112 such that the display area 112 is maintainedin synchronicity with the user's preferences as expressed (eitherexplicitly or inherently) through the interactions in the chat area 110.In this embodiment, utilized with a travel related chat drivenartificial intelligence system, the display area 112 includes a map area120 and a card display area 130. The map area 120 displays a geographicarea or one or more location points (e.g., destinations) associated withthe user preferences expressed through interactions with the chat area110, while the card display area 130 may display a number of displayobjects (i.e., “cards”), where each of the cards presents contentassociated with a particular entity (e.g., destination, flight, hotel,travel packages, etc.) associated with the user preferences expressedthough interactions with the chat area 110.

The card display area 130 may be a slider or carousel as is known in theart such that it may contain multiple cards, a subset of which may bedisplayed at a given point. The user may scroll or flip through thevarious cards by interacting with card display area 130. The geographicarea or location points displayed in map area 120 may be associated withthe cards in the card display area 130 such that the geographic areas orlocations associated with the cards in the card display area 130 may bedepicted on a map displayed in the map area 120. For example, in FIG.1A, the card 132 is associated with the destination “Cape Town” which isalso displayed in the map displayed in map area 120. For example, in theexample of FIG. 1A, the card 130 includes three images corresponding to“Phuket Province” and the card 132 includes three images correspondingto “Cape Town.” The images can be selected for display in any desiredmanner. Detailed examples of an image selection process are providedbelow.

Some embodiments of interface 100 may also have a static data entry area140. This area 140 may allow a user to enter static data for anitinerary or may reflect data selected by a user (e.g., clicking on aparticular destination card) that may include information such as datesor data ranges, number of travelers (e.g., adults or children) or aparticular specified destination. The data provided by a user in thestatic data entry area 140 may be used in association with theinteractions in the chat area 110 to drive content in the display area112. Conversely, data displayed in the static display area 140 may bedetermined by the chat driven artificial intelligence system frominteractions in the chat area 110. The static data entry area 140 maythus be populated by the chat driven artificial intelligence systembased on these determinations.

It will be noted that the embodiments depicted in FIGS. 1A-1D arepresented by way of example only. Other interfaces for other embodimentsof chat driven artificial intelligent systems are possible andcontemplated herein. For example, certain of these interfaces maypresent text or documents or a list of links of search results or othertypes of data based on interactions in the chat area or may displaycontent differently. Additionally, such chat driven artificialintelligence interfaces may be presented through other channels,including through third party web pages or applications, even when suchthird party web pages or applications may have proprietary interfaces.

Turning then to FIGS. 1E-1H, embodiments of an interface for a travelrelated chat driven artificial intelligence system that may be presentedthrough a third party web page, application or interface are depicted.Initially, a user may interact with an interface presented by the thirdparty (e.g., a Facebook feed or news page) as depicted in FIG. 1E. Alink 190 may be embedded in the third party interface, where the linkmay take the form of an advertisement or sponsored post by the operatorsof the chat driven artificial intelligence system. By accessing orinteracting with the link the user may be taken to a proprietaryinterface, platform or application operated by the third party (e.g.,Facebook Messenger or WhatsApp Messenger). A widget for the chat drivenartificial intelligence platform (e.g., an application or a “bot” suchas a Facebook Messenger bot) may execute within the context of the thirdparty platform (e.g., in the example illustrated Facebook Messenger).

For example in FIG. 1F, interface 150 may include a chat portion or area152 and a display area 154. The user interacts with the chat drivenartificial intelligence system through the chat area 152. As shown inFIG. 1G, the user's interactions in the chat area 152 may appear in thedisplay area 154, as is typical in messaging applications or the like.In particular, a human understandable natural language conversation mayoccur in the display area 154 based on interactions provided by the userin the chat portion 152 such that the user may iteratively provide aninteraction and the chat driven artificial intelligence system mayprovide an interaction prompt or a response interaction in the displayarea 154. These interactions can be exchanged between the user and thechat driven artificial intelligence system using the chat area 152 anddisplay area 154.

The interactions taking place in through the chat area 152 and displayarea 154 drive the display of content in display area 154 such that thedisplay area 154 is maintained in synchronicity with the user'spreferences as expressed (either explicitly or inherently) through theinteractions entered in the chat area 152. In this embodiment, utilizedwith a travel related chat driven artificial intelligence system, theinteractions presented through the display area 154 may include a cardor carousel display area 156 associated with the user preferencesexpressed through interactions with the chat area 152 including a numberof display objects (i.e., “cards”), where each of the cards presentscontent associated with a particular entity (e.g., destination, flight,hotel, travel packages, etc.) associated with the user preferencesexpressed though interactions with the chat area 152.

The card display area 156 may be a slider or carousel as is known in theart (in the case a card carousel as offered through the FacebookMessenger application) such that it may contain multiple cards, a subsetof which may be displayed at a given point. The user may scroll or flipthrough the various cards by interacting with card display area 156 asshown in FIG. 1H.

Now looking at FIGS. 1I and 1J, embodiments of an interface for a travelrelated chat driven artificial intelligence system that may be presentedin a container (e.g., an ad container) on a third party web page aredepicted. Here, interface 170 may be presented in a third party webpage, for example, by using an advertisement network infrastructure orby referencing an endpoint or location that provides the interface 170.Initially, a user may interact with an interface presented in the thirdparty web page as depicted in FIG. 1I. A widget for the chat drivenartificial intelligence platform may execute within the context of thecontainer on the third party web page.

For example in FIG. 1I, interface 170 may include a chat portion or area172 and a display area 174. The user interacts with the chat drivenartificial intelligence system through the chat area 172. As shown inFIG. 1J, the user may interact in the chat area 172. In particular, ahuman understandable natural language conversation may occur in the chatarea 172 based on interactions provided by the user such that the usermay iteratively provide an interaction and the chat driven artificialintelligence system may provide an interaction prompt or a responseinteraction in the chat area 172.

The interactions taking place in through the chat area 172 drive thedisplay of content in display area 174 such that the display area 174 ismaintained in synchronicity with the user's preferences as expressed(either explicitly or inherently) through the interactions entered inthe chat area 172. In this embodiment, utilized with a travel relatedchat driven artificial intelligence system, the interactions presentedthrough the display area 174 may include a card or carousel display area176 associated with the user preferences expressed through interactionswith the chat area 172 including a number of display objects (i.e.,“cards”), where each of the cards presents content associated with aparticular entity (e.g., destination, flight, hotel, travel packages,etc.) associated with the user preferences expressed though interactionswith the chat area 172. The user may scroll or flip through the variouscards by interacting with card display area 176.

With these embodiments of interfaces for a chat driven artificialintelligence system in mind, including interfaces for a travel relatedchat driven artificial intelligence system, embodiments of a chat drivenartificial intelligence systems that utilize and present such interfacesmay now be discussed. While such embodiments are presented in thecontext of travel related chat driven artificial intelligence systems,it will be understood that this context is given by way of example only,and other embodiments of such chat driven artificial intelligencesystems may be utilized in other contexts, such as real-estate or thelike.

Moving then to FIG. 2, one embodiment of a computer network topologythat includes an embodiment of such a chat driven artificialintelligence system (or “chat system”) is depicted. Chat system 200 maybe accessed over a network 230 such as the Internet, an intranet, aLocal Area Network (LAN), a Wide Area Network (WAN), a wireless orcellular network, a wireline network or some combination of networks) byusers through applications at computing devices 202. Chat system 200 maybe implemented or deployed, for example, on a cloud based platform suchas Amazon Web Services, Microsoft Azure, Google Cloud, etc.

To facilitate these accesses, a chat interface 210 of the chat system200 may provide an interface through which requests may be received andresponded to over network 230. This chat interface 210 may be anApplication Programming Interface (API) such as a Web services API,including, for example, a REpresentational State Transfer (REST orRESTful) API. In one specific embodiment, the chat interface 210 may beimplemented as a single endpoint RESTful API such that each request tothe chat interface 210 may be a call to this single interface thatincludes one or more query parameters. In another embodiment, the chatinterface 210 may be implemented as two endpoints where one of theendpoints corresponds to an initialization or “init” endpoint (e.g.,“address”/init) to set up the chat system widget at the user's device202 or to establish or restore a session, and a “chat” endpoint (e.g.,“address”/chat) for all other requests to the chat interface 210.

Requests to chat interface 210 are thus issued from user devices 202based on user's interactions with the chat driven interface 214 renderedat their device 202. These requests may be received by the chatinterface 210 and a response determined by the chat system 200 andprovided back to the requestor through the chat interface 210. In oneembodiment, for example, the chat system 200 may provide a web site,where that web site may be a standalone web site provided by the chatsystem 200 (e.g., through chat interface 210) or may be hosted as amicro site within another web site, such as a web site provided by apublisher from a publisher web server 204. These publishers may, forexample, be travel service providers such as airlines, hotel providers,etc. who wish to leverage the functionality of the chat system 200 toimprove the ability of users of their site to determine or locateofferings provided in association with that travel service provider.Thus, a user may visit a web site provided from the publisher (e.g.,through the publisher web server 204). By accessing a link or beingotherwise (re) directed or routed the user may enter a micro site (e.g.,that may have the look and feel of the publisher's web site) that isprovided by chat system 200.

Such a web site may therefore be accessed by the user at their device202 using a web browser 206 on their device. When the user access theweb site provided by the chat system 200 (e.g., as part of a micro siteaccessed from the web site provided by the publisher or as a standaloneweb site), a request is sent to the chat interface 210 which respondswith a web page (or portion thereof) that can be rendered in the user'sbrowser 206. In particular, when the web site provided by the chatsystem 200 is accessed, a chat system widget 208 may be provided inassociation with an interface returned to the user's browser 206 inresponse to a request (e.g., utilizing a script or embed tag). The chatsystem widget 208 may be, for example, JavaScript or other code that maybe executed by the user's browser 206 or by the user's device 202. Asthe user interacts with the chat driven interface 214 provided by thechat system 200 through the user's browser 206, the chat system widget208 may send requests to the chat interface 210 based upon the user'sinteraction with the chat driven interface 214. The chat system 200receives these requests through the chat interface 210 and respondsthrough the chat interface 210. These responses may include one or moreobjects (e.g., such as JavaScript Object Notation (JSON) objects). Thechat system widget 208 may receive these responses and process theincluded objects to render or update the chat driven interface 214 inthe browser 206.

In addition to being travel service providers such as airlines or hotelproviders, publishers may also be almost any other third party providinga web site such as magazine or newspaper web site providers (e.g., CondeNast, Travel and Leisure, Modern Bride, Surfing, etc.), blogs,retailers, etc. Thus, in addition to being provided as a micro site fora publisher, a chat driven interface 214 may be provided directly in theweb pages 212 provided by these publishers. Specifically, in certainembodiments, a chat driven interface 214 may be presented in a displaycontainer (or unit) on a publisher's web page 212 alongside the contentprovided by the publisher on the web page. The display container may beof the size typically used to display a web based advertisement such asa 600×300 size container (often referred to as an ad container or adunit).

Here, when the user access the web site provided by the publisher webserver 204 at their device 202 using a web browser 206 on their devicethe publisher's web page 212 may be provided by the publisher's webserver 204. A chat system widget 205 may be provided in association witha display container (e.g., a container defined by a div or iframe tag)in the publisher's web page 212. For example, the chat system widget 205may be incorporated into the publishers web page 212 or otherwiseprovided by publisher's web server 204, or may be requested from chatsystem 200 when the web page 212 is loaded by browser 206 (e.g., using ascript tag or the like). This chat system widget 205 may be JavaScriptor other code that may be executed by the user's browser 206 or by theuser's device 202. In one embodiment, the chat system widget 205 mayinclude, all of, or a subset of, the functionality of chat system widget208.

When the chat system widget 205 is loaded, it may send an initialrequest to the chat system identifying the publisher, the publisher'sweb page 212 or other data. The chat system 200 receives this requestthrough the chat interface 210. The chat system 200 utilizes the contextof the publisher's web page 212 (e.g., concepts or destinationsassociated with the web page 212) to select content (e.g., from anadvertiser) and form a response with this content and an initialinteraction to return to the chat system widget 205 through interface210. These responses may include one or more objects (e.g., JSONobjects) associated with the selected advertiser content. The chatsystem widget 205 may receive this response and process the includedobjects to render or update the chat driven interface 214 presented inthe container on the publisher's web page 212. Thus, the initial stateof the chat driven interface 214 presented on the publisher's web page212 is related to the context of the content on the publisher's web page212.

Subsequently, as the user interacts with the chat driven interface 214provided by the chat system 200 on the publisher's web page 212 (e.g.,has chat interactions with the chat portion of the interface), thewidget 205 may send requests to the chat interface 210 based upon theuser's interaction with the chat driven interface 214. The chat system200 receives these requests through the chat interface 210 and respondsthrough the chat interface 210. The chat system widget 205 may receivethese responses and process the included objects to render or update thechat driven interface 214 in the publisher's web page 212.

The chat interface 210 may also be used to provide an interface for astandalone application 207 (or “app”) that resides on a user's device202. This application 207 may be a standalone application affiliatedwith the chat system 200 such that when the application 207 is startedup or launched on the device 202 the application 207 is configured torender a chat driven interface 214 and contact the chat system 200through chat interface 210. In this instance, the application 207 isconfigured to send requests to the chat interface 210 based upon theuser's interaction with the chat driven interface 214 presented by theapplication 207. The chat system 200 receives these requests through thechat interface 210 and responds, where the response includes one or moreobjects (e.g., JSON objects). The application 207 may receive thisresponse and process the included objects to render or update the chatdriven interface 214 presented through the application 207.

In certain embodiments, the chat interface 210 may also be used toprovide an interface through a proprietary third party interface (e.g.,web page) or application that resides on a user's device 202.Specifically, in many instances certain proprietary web based platforms(including certain social media platforms like Facebook or WhatsApp) mayprovide proprietary applications that may be accessed through theirplatform. One example of such an application or platform is FacebookMessenger. Chat interface 210 may be used to provide chat driveninterfaces 214 through such proprietary platforms 216. In oneembodiment, then, a link or other access point to chat system widget 203may be provided through the proprietary platform. For example, in thecontext of a Facebook a sponsored post may be presented through a user'sfeed where the sponsored post may have a link or access point for chatsystem widget 203. Here, in one embodiment, chat system widget 203 maybe a Facebook Messenger bot that may operate in the Facebook Messengerenvironment.

Thus, the chat system widget 203 may render chat driven interface 214 inthe proprietary platform 216 and may utilize functionality ofproprietary platform 216 (e.g., messaging functionality or the like), inconjunction with, or as part of, the chat driven interface. It may berealized, that in many cases, operators or providers of proprietaryplatforms 290 do not allow direct communications to or from otherentities in the environment of their proprietary platform. Thus, in manycases, all communications within such a proprietary platform may passthrough proprietary platform provider 290. Thus, for example, in thecontext of Facebook Messenger, communications pertaining to interactionswith or other activity of the user with respect to chat system widget203 and chat driven interface 214 may be communicated from proprietaryplatform provider 290.

Moreover, all communications to chat system widget 203 and chat driveninterface 214 may have to originate from proprietary platform provider290 and be formatted for consumption or rendering by proprietaryplatform 216. In order to operate in such an environment then, chatsystem 200 may include proprietary interface 280. Chat system widget 203is adapted to issue calls to proprietary platform provider 290 thatidentify an endpoint associated with proprietary interface and which mayidentify a user, a message (e.g., a chat interaction) or other payloaddata that may identify other interactions. In response to this call,proprietary platform provider 290 may issue a call to the identifiedendpoint at proprietary interface 280 that identifies the user andincludes the message (e.g., chat input) that the user provided in themessenger based chat interface. Such a call can also include a custompayload identifying other interactions with the chat driven interface214, such as a button press or the like. The call from the proprietaryplatform provider 290 may utilize a globally unique identifier (GUID)that is unique to the proprietary platform provider 290.

Proprietary interface 280 may be adapted to receive communications fromproprietary platform provider 290 and store or access proprietaryinterface session data 282 that comprises data associated with callsfrom proprietary platform provider 290 keyed by the GUID provided byproprietary platform provider 290. Thus, proprietary interface sessiondata 282 includes data provided by the proprietary platform provider 290and data determined or returned by the chat system 200 associated withthe GUID provided by proprietary platform provider 290. When theproprietary interface 280 receives a call from proprietary platformprovider 290, the proprietary interface 280 can use the GUID of the callto access the proprietary interface session data 282. Using any locatedproprietary interface session data 282 and the data in the call receivedfrom the proprietary platform provider 290, the proprietary interface280 may form a request to the chat interface 210 in the form, and withparameters, expected by the chat interface 210.

These requests are received through the chat interface 210 and the chatsystem 200 determines a response that is provided back to theproprietary interface 280, where the response includes one or moreobjects (e.g., JSON objects). The proprietary interface 280 is adaptedto take the response from the chat interface 210 and translate or formatthe response into a format expected by the proprietary platform 216(e.g., for example, according to the Facebook Messenger API). Thisresponse can then be provided back to proprietary platform provider 290which, in turn, provides it to proprietary platform 216 where it can beused to render or update the chat driven interface 214.

Thus, requests may be received at the chat interface 210 from chatsystem widgets 205, 208, proprietary interface 280, or application 207and be processed by the chat system 200 to determine a response toreturn to the requesting widget 205, 208, proprietary interface 280, orapplication 207. The chat system widget 205, 208, proprietary interface280, or application 207 thus sends a request to the chat interface basedon a user's interaction with the chat portion of the chat driveninterface 210. This request may therefore include, the unstructured ornatural language (NL) data submitted by the user through the chatportion of the chat interface (e.g., a statement or question that may,or not, be properly formed language). The request may also includeparameters having static or structured data associated with the user'sinteraction, such as data provided by the user through a static portionof the chat driven interface 214, data on other events associated withthe chat driven interface 214 such as what portions of the page the userinteracted with or clicked on (e.g., tracking data). The request mayalso include an identifier, such as a globally unique identifier (GUID)for the user as used by the chat system 200. Such an identifier may bestored in cookie data on the device 202 as is known in the art.

Chat system 200 may store historical session data 272 associated withthe user in addition to current session data 274 associated with therequesting user. Historical session data 272 may include data onsessions that occurred with the user in the past (e.g., outside of sometime window), while the current session 274 for the user may includedata associated with requests that have been received within aparticular most recent time window or that have taken place without thelapse of a certain amount of time. Each of the historical sessions 272or current session 274 may include a collection of frames associatedwith the user where the frames are associated with requests receivedfrom that user during the corresponding session.

Each frame of a session (e.g., a historical session 272 or the currentsession 274) corresponds to the data associated with responding to aparticular request received through the chat interface 210 during thesession. A frame may thus include, the received request itself, datadetermined from the requests such as the parameters of the request,including the user interaction(s) with the chat system interface thatgenerated the request, data determined based on the request and dataincluded in the response. A frame may also include one or more JSONobjects that were included in, or used to determine, the JSON objectsincluded in the response to the received request. These JSON objects maybe nested or may include one or more arrays of values.

As discussed the chat interface 210 may be a single endpoint or may atleast two endpoints, thus if the chat interface is a single endpoint therequest may be initially received at the chat interface 210 and passedto orchestrator 240. If the chat interface 210 implements two or moreendpoints, one of the endpoints may correspond to an initialization or“init” endpoint to set up the chat system widget at the user's device202 or to establish or restore a session, and a query endpoint for allother requests to the chat interface 210. Thus, at this point, if therequest is an initialization request (e.g., at the init endpoint) atemplate 278 may be returned in response to the request. The templates278 may be identified and accessed based on the identity of the contextof the web page on which the chat driven interface 214 is to appear.Such an identity may be determined based on a template or otheridentifier included in the request received at the endpoint. Thetemplate may contain initialization configuration for the chat driveninterface 214 such as the style, branding, colors, initial context,initial prompt, seed phrase or data for the chat portion of aninterface, or other configuration of the chat driven interface.

In any event, when a request is received at chat interface 210 for anendpoint other than an init endpoint and passed to the orchestrator 240,the orchestrator 240 may obtain the user identifier from the receivedrequest and use the user identifier to obtain the historical sessiondata 272 (if any) associated with the user identifier and the currentsession data 274 (if any) associated with the user identifier. In oneembodiment, the historical session data 272 or current session data 274may be stored in easily accessible database such as Redis such that thehistorical session data 272 or current session data 274 may be obtainedas a value in the database associated with the key of the useridentifier. If these is no current session data 274 associated with theuser identifier, an entry in the data storage (e.g., Redis) associatedwith the user identifier received in the request may be created suchthat the data for the current session with the user may be stored.

The orchestrator 240 then processes the received request to generate aresponse to return. In one embodiment, the orchestrator 240 creates aworking frame 276 associated with the user's request and populates theworking frame 276 with the data as determined from the received request,including the user's interaction (e.g., natural language statement orchat interaction that caused the request to be generated), relevant datafrom previous frames in the current or historical sessions, the useridentifier, etc. The orchestrator 240 can then process the user's chatinteraction in the received request to extract the entities (includingconcepts or destinations) and intents associated with the user'sinteraction. This user interaction may be a natural language statement.Accordingly, the orchestrator 240 may employ one or more data extractionservices to extract these entities and intents. Generally, theseextraction services take as input a statement and apply natural languageprocessing techniques to extract one or more entities or intents toreturn.

Specifically, in the embodiment depicted, extraction services 242 mayinclude a set of services implemented on the chat system 200 to take asinput a statement (e.g., a natural language statement), extract one ormore entities (entity types) or intents from the statement and returnthe extracted entities or intents. These entities may include a set ofconcepts, including, for example, those related to geography orgeographic regions, destinations, time, trip types, activities, amounts,units, names, colors, etc. In one embodiment, for example, entities maybe travel related and include: Destination, AirportCodes, HotelAction,Hotel Brands, HotelsCombinedLocations, HotelStarRating, LocationType,Lodging, TravelActivities, Traveler and Triptype.

Intents may be classes that a statement (e.g., a natural languagestatement) falls into and that indicate how the chat system 200 shouldrespond to a user interaction. These intents may include, finddestination, find hotel, help, small talk, modify and replace or others.In particular, intents may be travel related and includeAnsweringFollowUpQuestion, Book Hotel, ContactCallCenter, DefaultFallback Intent, Default Welcome Intent, Email The User,EventViewingHotel, Find A Destination, Find A Hotel, Find a Package,FollowUpQuestion, Get Bots Name, Get Users Name, Greetings, HotelAmenity Checker, Instructions, Modify (Add) Destination Search, Modify(Add) Hotel Search, Modify (Replace) Destination Search, Modify(Replace) Hotel Search and Out of Scope Add follow-up intent.

These extraction services 242 may be particular to entities or intentsor even specific types of entities or intents and be configured to onlyextract that type of entity or intent from the received statement. Forexample, an extraction service 242 may be configured to extract onlytime. Such an extraction service may employ, for example, the StanfordTemporal Tagger (SuTime) or Duckling. Another extraction service 242 maybe a trip type classifier that extracts a trip type from the user'snatural language statement (e.g., sole, group, couples, romantic,buddies trip, etc.). Yet another extraction service 242 may beconfigured to extract just concepts or just intents. Other extractionservices 242 are possible and are fully contemplated herein.

Certain extraction services that chat system 200 utilizes may beimplemented by third party providers. Extraction services interface 244may allow such third party entity extraction services (not shown) to bequeried. In particular, the orchestrator 240 can send a requestcontaining a received user interaction (e.g., a natural languagestatement received through the chat portion of the interface) and anyother desired data associated with the request to these third partyextraction services through the extraction services interface 244. Anyentities or intents (or other data such as a natural language response)extracted or determined by that third party entity extraction servicecan be received back through the extraction services interface 244 atorchestrator 240. In some embodiments, a third party extraction servicemay also return a natural language response to the user's interaction.This natural language response may be associated with an intentdetermined for the user's interaction. Examples of such third partyextraction interfaces include, for example API.ai, IBM WatsonConversation, Google's Dialogflow or Amazon's Lex.

The data returned from each of the entity extraction services (e.g.,either entity extraction services 242 or third party entity extractionservices) may be saved into the working frame 276 associated with thecurrent request. Once the orchestrator 240 has the set of entities(including concepts) and intents associated with the user's interactionas determined from the initially received request, the orchestrator 240may use the entities (e.g., concepts) or intents as determined from theuser interaction by the entity extraction services to determine contentto provide in response to the request. In particular, the orchestratormay provide extracted concepts or intents to one or more data services.These data services may provide one or more travel related items (e.g.,destinations, regions, hotels, etc.) in response, along with contentrelated to those travel related items. The content may include, forexample, image data or links for the travel related item, locationrelated data, a description or ratings and review data for the travelrelated item, pricing data for the travel related item, or other typesof content. This content (or a subset thereof) can then be provided bythe orchestrator 240 in response to the initially received requestthrough chat interface 210.

Data services 250 may include a set of services implemented on the chatsystem 200 to take as input an entity (such as a concept) and accessconcept data 252 of the chat system 200 to determine related data. Theconcept data 252 may associate travel related items such asdestinations, regions, hotels with concepts. The data services 250 maythus take the concepts as inputs and use the concept data 252 to rank aset of travel related items and return the content for the top rankedset of travel related items. Thus, for example, a hotel data service 250may return particular hotels associated with a concept (e.g. the contentfor those hotels) or may return particular hotels associated with a setof concepts and a region, etc. As another example, a destination dataservice 250 may return particular destinations associated with a conceptor may return particular destinations associated with a set of conceptsand a region, etc.

As with the extractions services discussed above, one or more dataservices that it is desired to utilize may be implemented by third partyproviders. Data services interface 246 may allow such third party dataservices to be queried. In particular, the orchestrator 240 may sendrequests including concepts to these third party data services throughthe data services interface 246. Any travel related items extracted bythat third party entity data service can be received back through thedata services interface 246 at orchestrator 240. An, example of suchdata services may, for example, include a weather data service.

The data (e.g., the travel related items, including content associatedwith the travel related items) returned from each of the data services(e.g., either data services 244 or third party data services) may besaved into the working frame 276 associated with the current request.Again, the data associated with each travel related item returned fromeach data service may be stored in the working frame 276 as JSON objectsthat may be nested or may include one or more arrays of values.

The orchestrator 240 can then process or manipulate the working frame276 to create a response to the request from the chat system widget 205,208, proprietary interface 280, or application 207, where the responseincludes a return frame having a subset of the travel related itemsreturned by the data services and a responsive interaction. Inparticular, the orchestrator 240 may select one or more travel relateditems (and associated content) from the working frame 276 for includingin the return frame for the user. The selection may be based on theentities (including concepts) and intents expressed in the currentworking frame 276, the current session 274 associated with the user,historical sessions 272 associated with the user or data returned by adata service.

In one embodiment, a publisher interface 248 may include one or moreinterfaces through which a publisher provided service may be accessedwhere that service can determine one or more offerings of travel relateditems provided by the publisher (e.g., the publisher providing thepublisher web site) or data (e.g., pricing data) associated with thepublisher's offerings. These offerings may include, for example, flightsor hotels. Thus, the orchestrator may utilize this interface 248 todetermine if the publisher has travel related offerings corresponding tothe travel related items as determined from the data services and storedin the working frame 276, or pricing related to travel related itemsstored in the working frame 276. The availability or pricing of travelrelated offerings from the publisher may be utilized to select thetravel related items to include in the response frame for the user. Forexample, if the travel related items included in the working frameinclude multiple destinations, only those destination served by flightsfrom the publisher may be selected for inclusion in the response frame.Similarly, if the travel related items included in the working frameinclude multiple hotels, only those hotels affiliated or provided by thepublisher may be selected for inclusion in the response frame.

Once the travel items and responsive interaction are selected they maybe formatted into the response frame (e.g., as one or more JSON objectsand associated arrays) and included in a response returned to the userreturned through chat interface 210 (or returned to the proprietaryinterface 280 as discussed). Additionally, the working frame 276utilized by the orchestrator in the creation of this response may besaved into the current session data 274 associated with the user.

When the chat system widget 205, 208 or application 207 at the user'scomputing device 202 receives this response, it may render the responseto update the chat driven interface 214 presented to the user.Alternatively, as discussed, the proprietary interface 280 may take theresponse from the chat interface 210 and translate or format theresponse into a format expected by the proprietary platform 216. Thisresponse can then be provided back to proprietary platform provider 290which, in turn, provides it to proprietary platform 216 where it can beused to render or update the chat driven interface 214.

In particular, the chat system widget 205, 208, application 207 or chatsystem widget 203 in association with proprietary platform 216 maypresent the response interaction (e.g., a natural language or textstatement included in the response) in the chat portion of the chatdriven interface 214. Additionally, the chat system widget 205, 208,application 207 or chat system widget 203 in association withproprietary platform 216, may render each of the travel related items(e.g., each JSON object associated with a travel related item) as a cardto present in a card display area of the chat driven interface 214. Eachof these travel related items may also have map data (e.g., map datapoints such as coordinate data such as latitude or longitude, etc.).Chat system widget 205, 208 or application 207 may update a mappresented in the map display area of the chat driven interface 214 todisplay these travel related items in the map. In one embodiment, themap display area of the chat drive interface 214 may be provided by athird party provider such a Google Maps or MapBox. In this case, chatsystem widget 205, 208 or application 207 may call the third partyprovider with the map data included in the response and render the datareturned by the third party provider in the map display area of the chatdriven interface 214.

As both the responsive interaction presented in the chat portion of theinterface 214 and the content presented in the display area of the chatdriven interface 214 are based on the context of the humanunderstandable interactions in the chat potion of the interface 214, thecontent presented in the display portion of the interface 214 reflectsthe intents or concepts expressed in the chat portion of the interface214 and the chat portion of the interface 214 and the display portion ofthe interface 214 are kept in synchronicity. Moreover, as the usercontinues to interact with the chat portion of the chat system interface214 the content displayed in the interface 214 can also be refined orupdated in an ongoing manner by the chat system 200 based on theconcepts or intents expressed in a human understandable and comfortablemanner through these user interactions.

Looking now at FIG. 3, a more detailed depiction of one embodiment ofthe architecture of an orchestrator for use with embodiments of a chatdriven artificial intelligence system is presented. It will be recalledfrom the above discussion that requests to a chat interface of a chatsystem may be issued from a user device based on a user's interactionswith a chat driven interface of the chat system rendered at theirdevice, including the user's human understandable natural languageinteractions with a chat portion of the interface. In particularembodiments, a chat system widget is executing on the user's device orbrowser in association with the chat driven interface. As the userinteracts with the chat portion of the chat driven interface at hisdevice (e.g., by typing in a natural language query or statement), thechat system widget system sends a request (or call) to the chatinterface of the chat system.

This request may include the unstructured or natural language query orstatement (referred to generally as a statement) submitted by the userthrough the chat portion of the chat interface. The request may alsoinclude parameters having static or structured data associated with theuser's interaction, such as data provided by the user through a staticportion of the chat driven interface, data on other events associatedwith the chat driven interface such as what portions of the page theuser interacted with or clicked on (e.g., tracking data). The requestmay also include an identifier, such as a GUID, for the user. As thechat interface may provide a single endpoint for requests (e.g., fromchat system widgets, browsers, etc.), the data of the request may beincluded as query parameters of the request received at the chatinterface.

When the request is received at the chat system through the chat systeminterface it may be provided to orchestrator 300. When the request isreceived at the orchestrator it may initially be provided to the sessionretrieval module 302. The session retrieval module 302 may obtain theuser identifier (e.g., the GUID) from the received request and use theuser identifier to obtain historical session data (if any) associatedwith the user identifier and current session data (if any) associatedwith the user identifier. In one embodiment, the historical session dataor current session data may be stored in easily accessible database 358such as Redis such that the historical session data or current sessiondata may be obtained as a value in the database using the useridentifier as a key. If these is no current session data associated withthe user identifier, an entry in the data storage (e.g., Redis)associated with the user identifier received in the request may becreated such that the data for the current session with the user may bestored.

In one embodiment, this session data (e.g., historical session data orcurrent session data) may include one or more objects. These objects mayinclude, for example, XML objects, JSON objects or other types ofobjects. In particular, one embodiment that session data includes JSONobjects that may include other JSON objects, which each might include aset of arrays. These JSON objects for the session data may includeprevious working frames used in replying to a user's previous requestsor other data associated the user or the user's sessions. Thus, thesession data for the user may include the statements submitted by theuser during previous interactions with the chat portion of the chatsystem interface, the entities (including concepts or destinations) andintents associated with those interactions, the responsive interactions(e.g., responsive statements determined or returned to the user), thetravel related items determined for the user in processing thoseprevious interactions, or other data associated with the user orresponding to a user's previous interactions.

The received request, including the query parameters and the sessiondata obtained for the user, can then be provided to the NLP extractionmodule 304. The NLP extraction module 304 includes an NLP pre-processingmodule 306 that initially receives the request and the session data. TheNLP pre-processing module 306 creates a working frame 320, which may bea JSON object, extracts the user statement (e.g., the unstructurednatural language statement the user provided through the chat portion ofthe chat system interface) and stores the user statement in the workingframe 320. Any other parameters of the user request (e.g., structureddata, tracking data, the user's GUID, etc.) may also be stored in theworking frame 320.

The NLP pre-processing module 306 can then determine if the requestincludes valid data such that it can be processed. This may includeevaluating the user statement for content (e.g., if the user's statementis empty it is not valid) or checking any supplied dates, destinationsof price ranges for valid ranges (e.g., within the next year, etc.). Ifthe request is not valid a default responsive chat interaction (e.g.,“I'm sorry, I didn't understand that,” “Can you tell me some more?”,etc.) may be stored as the response interaction in the working frame320. Alternatively, the follow up module 312 may be provided with theworking frame 320 which may determine and include such a defaultresponsive chat interaction in the working frame 320.

If the request is valid, the NLP extractor 308 may send requests to aset of extraction services to extract entities and intents from the userstatement. Generally, these extraction services are NLP extractionservices that take as input a statement and apply natural languageprocessing techniques to extract one or more entities or intents toreturn. In particular, the extraction services may include a set ofinternal services 334 implemented on the chat system or a set ofexternal services 332 implemented by third parties that can receiverequests including the statement (e.g., the user's natural languagestatement), extract one or more entities or intents from the statementand return the extracted entities or intent.

In particular, the NLP extractor 308 can send a request containing theuser statement and any other desired data associated with the request tothese extraction services through an interface 309. As a default, theuser statement may be sent to each of the NLP extraction services 332,334, however, in other embodiments a subset of these NLP extractionservices 332, 334 may be determined and utilized. For example, only theinternal services 334 may be utilized or only the external services 332may be utilized. Alternatively, a combination of internal services 334and external services 332 may be used. For example, in one embodiment,external NLP services 332 may be utilized for processing any “smalltalk”, while an internal NLP service 334 may be used for intents,concepts or entities.

The NLP extractor 308 may be configured to form a request to eachextraction service according to a format utilized by that particularextraction service. For a particular extraction service, then, the NLPextractor 308 may populate the request with appropriate data from theworking frame 320, including the user statement, the user GUID or otheruser session data if needed, or other data that may be used in a requestto the extraction service 332, 334. The availability of certain data maydetermine whether a particular extraction service 332, 334 may beutilized. For example, if certain data needed to form a request to aparticular extraction service 332, 334 is not available in the workingframe 320 that particular extraction service 332, 334 may not beutilized.

In one embodiment, the NLP extractor 308 may have a different functionfor formulating API requests to each extraction service 332, 334. TheNLP extractor 308 may utilize a set of extraction service rules todetermine the extraction services to utilize and each of the functionsfor the determined extraction services 332, 334 may extract data fromthe working frame 320 to populate a request for the correspondingextraction service 332, 334. If a function cannot populate acorresponding request to the extraction service, that extraction service332, 334 may not be utilized. Functions associated with certainextraction services 332, 334 that cannot be populate a correspondingrequest may trigger a response interaction requesting additionalinformation in the follow up module 312 as will be discussed in moredetail. Such a flag or response interaction may be placed in workingframe 320.

Any entities or intents (or other data such as a natural languageresponse to return) extracted or determined by an entity extractionservice 332, 334 can be received back through the interface 309 at NLPextractor 308 and stored in the working frame 320 along with anidentifier of the extraction service 332, 334 that returned that data.Additionally, the values returned by the extraction services 332, 334may be normalized by NLP extractor 308 before they are stored in theworking frame. For example, destinations or regions may be normalizedfor use by the chat system.

In particular, the return from an entity extraction service 332, 334 mayinclude a JSON object with entities (e.g., values for specified entitytypes), intents or a responsive interaction. These returned JSON objectsmay be stored as nested JSON objects in the working frame 320. Theentities may include a set of concepts, including, for example, thoserelated to geography or geographic regions, destinations, time, triptypes, activities, amounts, units, names, colors, etc. For example,travel related entity types may include: Destination, AirportCodes,HotelAction, HotelBrands, HotelsCombinedLocations, HotelStarRating,LocationType, Lodging, TravelActivities, Traveler and Triptype.

Intents may be classes that a statement (e.g., a natural languagestatement) falls into and that indicate how the chat system shouldrespond to a user interaction. These intents may include, finddestination, find hotel, help, small talk, modify and replace, FAQ,Email The User or others. Specifically, travel related intents mayinclude AnsweringFollowUpQuestion, Book Hotel, ContactCallCenter,Default Fallback Intent, Default Welcome Intent, EventViewingHotel, FindA Destination, Find A Hotel, Find a Package, FollowUpQuestion, Get BotsName, Get Users Name, Greetings, Hotel Amenity Checker, Instructions,Modify (Add) Destination Search, Modify (Add) Hotel Search, Modify(Replace) Destination Search, Modify (Replace) Hotel Search and Out ofScope Add follow-up intent. In another embodiment, intents may includeFind A Destination, Find A Hotel, Small Talk, Terms of Service, FAQ,Hotel Amenities, Book a Hotel, and Ask For an Agent.

The extraction services 332, 334 may include a number of extractionservices 332, 334 that may be general services for extracting entities(e.g., concepts) and intents from natural language statements or thatmay be specific services for extracting only entities or intents orspecific types of entities or intents. In one embodiment, a conversationmanagement or interface platform may be used as an extraction service.Examples of such conversation management or interface platforms includeAPI.ai, IBM's Watson Conversation, Wt.ai., Converse.ai, etc.

In the main, these conversation management platforms function usingmachine learning trained on a set of intents or entities defined by theoperators of the travel related chat system such that when a requestincluding the user's natural language statement is submitted to theconversation platform the statement can be classified as having aparticular intent and having values for a set of entities. These intentsand entities may then be returned to the requestor by the conversationplatform. The travel related chat system can thus use the conversationplatform to extract intents and entities specified and defined byoperators of the travel related chat system. Moreover, the conversationplatform may be trained or otherwise configured to provide a responsiveinteraction associated with the intent into which the user's statementis classified. Thus, in addition to the intents and entities returned bysuch a conversation platform, a responsive interaction (e.g., a textstatement to present to a user) may also be returned by a conversationmanagement platform.

As but one example, in internal NLP extraction service 334 may betrained by providing a machine learning NLP tool with a set of trainingdata annotated using the brat rapid annotation tool (BRAT). Otherextraction services 332, 334 may include entity extraction services 332,334 for extracting only entities or only concepts from natural languagestatements or an intent extraction service for extracting only intentsfrom natural language statements. One extraction service may beconfigured to extract a TripType entity from a natural languagestatement. Another entity extraction service may be configured toextract and return a destination or geographic identifier (e.g., such asa region) from a natural language statements. To facilitate theextraction of geographic identifiers, an NLP extraction service 334 mayemploy a knowledge base 358 that correlates regions with points ofinterest or destinations. Thus, given points of interest, destinations,or regions, associated regions, points of interest or destinations maybe determined. Other extraction services, interfaces and formats thereofare possible and are fully contemplated herein.

In order to further improve performance by reducing bandwidth usage ortime to respond to a user interaction through the chat system interface,a caching framework may be employed between the NLP extractor 308 andthe extraction services 332, 334. This caching framework may serve tocache requests to external extraction services so responses can beaccessed without submitting the request to the extraction serviceitself. This cache framework can be keyed based on request data in orderthat the same request retrieves the response originally returned fromthe extraction service in response to a previously identical requestfrom the cache.

Once the response are received from the extraction services 332, 334 andstored in the working frame 320, NLP post processing module 310 maydetermine canonical values for any different values for entity types orintents returned by different extraction services. The values for anentity type or intent selected as canonical values may be identified inthe working frame.

In particular, one embodiment, the NLP post processing module 310 mayinclude a set of extraction service rules 350 defining a hierarchy oftrust for the set of entity extraction services. More specifically, inone embodiment, the set of extraction service rules 350 may define ahierarchy of trust associated with each entity type, defining for eachentity type an order of extraction services, such that if multipleextraction services return values for an entity type, the rule may beutilized to select as the canonical value for that entity type the valuereturned from the highest ranked extraction service associated with thatentity type. An intent rule may define a hierarchy of trust associatedwith each extraction service such that an intent returned by a higherordered extraction service may be selected as a canonical intent beforean intent returned by a lower ordered extraction service.

NLP post processing module 310 may thus process working frame 320 todetermine if any entity type has different values returned by differentextraction services for the user statement or different extractionservices returned different intents for the user statement and apply oneor more extraction services rules 350 to select canonical values for theentity types or intents. Extraction service rules 350 may also specifythat an intent or value for entity type from a particular extractionservice is to be utilized as a canonical value (or not utilized at alleven if no other values for the entity type have been returned)regardless of values returned by other extraction services. Theseextraction services rules 350 may, for example, be implemented asmachine learning based rules that may be refined over time.

For example, one extraction service may return a value of “romantic” forthe entity type TripType while another extraction service may return avalue of “business trip” for the entity type TripType. If the extractionservice rule 350 associated with the entity type TripType orders thefirst extraction service higher than the second extraction service, thevalue “romantic” may be selected as a canonical value for the entityTripType. Once the canonical values for each entity type or intent areselected for any duplicative entity types or different intents.

Once the entities and intents (or other data) returned by the extractionservices 322, 324 are stored in the working frame 320 and any canonicalvalues for entities or intents identified in the working frame 320, theNLP post processing module 310 may determine if any data any of theuser's previous frames in the current session should be included inworking frame 320. If there is no previous frame in the current session(i.e. the user statement is the first user statement of the session) nofurther action is taken. If, however, one or more previous frame existfor the user in the current session (e.g., the user has been interactingwith the chat portion of the chat system interface and the chat systeminterface has been updated in response to the those user interactions),the intent as determined for the current request (as stored in workingframe 320) may be used to determine whether, and how, to populate theworking frame 320 with any data from any previous frames of the currentsession.

In particular, a set of intent evaluation rules 352 may be used topopulate the working frame 320 with data from a previous frame (ifneeded). For example, the intent as returned by the extraction service332, 334 may be compared with one or more intents stored in the workingframe to determine an intent to utilize (e.g., a canonical intent forthe working frame). As an example of other intent evaluation rules 352,one rule may specify that a canonical intent may be aHotelAmenityChecker or BookHotel intent if a user has previously beenshown hotels, otherwise any intent expressed in a current interactionmay be overridden and the intent HotelSearch determined as a canonicalintent and stored in the working frame 332. As another example of aintent evaluation rule, if any points of interest are extracted anyintent expressed in a current interaction may be overridden and theintent HotelSearch determined as a canonical intent and stored in theworking frame 332. As yet another example of a rule, if any explicitlyprovided search frame data (e.g., destinations/stars/travelers/concepts)can be extracted, an intent for a previous type of search may bedetermined as a canonical intent and stored in the working frame 332,regardless of any other non-search intents that may be extracted fromthe current interaction.

As another example, if the intent as stored in the working frame 320 isa NewDestinationSearch the values for entities as extracted by theextraction services 332, 334 may remain in the working frame 320. If theintent as stored in the working frame 320 is a ModifyDestinationReplacethe entities from the immediately previous frame of the session may beused to populate the working frame (e.g., they may be copied into theworking frame) except for the entity type of destination. For thatentity type (destination), the value as returned by the extractionservices may be kept and utilized in the working frame 320. If theintent as stored in the working frame 320 is aModifyDestinationAdditive, the entities from the immediately previousframe of the session may be used to populate the working frame 320(e.g., they may be copied into the working frame) including the valuesfor the entity type of destination. Additionally, the value as returnedby the extraction services for the entity type destination may be addedto the values for the destination entity copied from the previousframe(s). Similar rules may also be applied to entity types related toconcept, date, price, duration, NumberOfTravelers, or others.

Other rules may be pertinent to the environment in which a chat driveninterface is utilized. For example, if the chat driven interface is usedin association with a particular partner (e.g., enterprise) utilizingthe chat system, these rules may be specific to that partner. Forexample, certain rules may evaluate any returned airport to determine ifit is an airport served by the partner. Another rule may evaluate anyreturned regions to determine if one or more, or all, of the regionsare, or are not, served by the partner, while another rule may check thespecified dates to determine if the user is utilized dates that are toonear or far in time. If any of these e partner specific rules areviolated, an exception flag may be set in the working frame 320.

It will be understood that these rules are provided by way of example,other rules will be possible and are fully contemplated herein.Additionally, other techniques for the population of the working frame320 with data from one or more previous frames of the session may alsobe utilized. For example, a supervised or unsupervised machine learningmodel may be utilized to determine what data to include (or replace) inthe working frame 320 from previous frames in the current session of theuser.

At this point then, working frame 320 may include at least a set ofentities (e.g., entry types and value) and intents returned from theextraction services 332, 334, an identification of canonical values foreach entity type or intent returned by the extraction services 332, 334and, in some embodiments, at least one responsive interaction (e.g.,natural language response to the user' interaction) determined by anextraction service 332, 334. The identification of the canonical valuesmay include an identification of one or more most recent intents locatedfrom the most recently received user interaction.

The orchestrator 300 may then apply intent processing module 314 to theworking frame to determine content to provide in response to the requestreceived from the user based on entities (e.g., concepts) or intents asdetermined from the user interaction by the entity extraction servicesand stored in working frame 320. The intent processing module 314 maydetermine the intent (e.g., the canonical intent) associated with thereceived request as stored in the working frame 320. Using intenthandling rules 368 specifying an action associated with an intent, theintent processing module 314 may pass control to the post-processing andresponse formatting module 316, determine a responsive interaction toinclude in the working frame 320 or determine one or more data servicesto call and what data to provide to each of the data services.

For example, if the intent is ContactCallCenter an intent handling rule368 may have an action to include the call center telephone number in aresponsive interaction to the user and store the responsive interactionin the working frame 320. If, however, the intent is Find A Destination,Find A Hotel, Find a Package, Modify (Add) Destination Search, Modify(Add) Hotel Search, Modify (Replace) Destination Search, Modify(Replace) Hotel Search the intent handling rules 368 may specify whichdata service 370 to call and what data from the working frame 320 toprovide in the request to the data service 370.

In particular, the orchestrator 300 may provide extracted entities(e.g., concepts) or intents from the working frame 320 to one or more ofthe data services 370. Specifically, the intent processing module 314may determine one or more data services 370 to call based on a canonicalintent expressed in the working frame 320 and may form a request to thedata service 370 that includes a canonical entity (e.g., canonicalconcept) determined for the working frame 320. These data services 370may provide one or more travel related items (e.g., destinations,regions, hotels, etc.) in response, along with content related to thosetravel related items. The content may include, for example, objects withimage data or links for the travel related item, location related data,a description or ratings and review data for the travel related item,pricing data for the travel related item, or other types of content.

In certain embodiments, data services 370 may include a set of services(implemented on the chat system or by external third parties) to take asinput an entity (such as a concept or destination) and access entitydata related to the entity (e.g., concept or destination) to returncontent associated with the entity. The entity data may associate travelrelated items such as destinations, regions or hotels with concepts ordestinations. The data service 370 can then respond to the request withcontent associated with content related to the input entity (e.g.,concepts or destinations). Thus, for example, a hotel data service 370may return particular hotels associated with a concept or may returnparticular hotels associated with a set of concepts and a region, etc.

Specifically, in one embodiment, a hotel service 370 may take anidentifier of a publisher or partner (e.g., advertiser) with whom thechat system is being utilized (which may be included in the initiallyreceived request or working frame 320), a region or a set of concepts.The hotel service 370 may then determine the hotels to return (e.g.,those hotels associated with the publisher or which the publisherserves, hotels associated with an advertiser, etc.) based on thepublisher identifier, the region or the set of concepts. The hotelservice 370 may return a set of hotel objects (e.g., JSON objects) withhotel name, images, concepts, scores, etc. Each of the hotel objectsreturned by the hotel service 370 may get stored in the working frame320.

As another example, a destination data service 370 may return particulardestinations associated with region, destinations associated with aconcept or may return particular destinations associated with a set ofconcepts and a region, etc. In particular, in one embodiment, adestination service 370 may take an identifier of a publisher or partner(e.g., advertiser) with whom the chat system is being utilized (whichmay be included in the initially received request or working frame 320),a region or a set of concepts. The destination service 370 may thendetermine the destinations to return (e.g., those destinationsassociated with the publisher or which the publisher serves ordestinations associated with an advertiser) based on the publisheridentifier, the region or the set of concepts. The destination service370 may return a set of destination objects (e.g., JSON objects) withdestination name, images, concepts scores, etc. Each of the destinationobjects returned by the destination service 370 may get stored in theworking frame 320.

After the intents have been processed by the intent processing module314, the working frame 320 now has a set of objects (e.g., destinationobjects or content objects) that may be presented as content to the useralong and in some cases may have a responsive interaction to present tothe user (e.g., that was determined by an extractor service 332, 334such as in the case of a Small Talk intent).

Follow up module 312 may then evaluate the entities and intents of theworking frame to determine what responsive interaction, if any, shouldbe returned as a responsive interaction to the user. This determinationmay include a determination whether more information is needed from theuser or if a follow up question should be returned as a responsiveinteraction to the user. Such a follow up question may be desired to,for example, obtain additional information from the user to augment orconduct a search for content to return. For example, this type ofinformation may be desired, if the working frame 320 does not includedata on time or date ranges, possible amount to spend, possible regionsor destinations, etc.

In particular, the follow up module 312 may include a set of follow uprules 366 for determining if a follow up question (e.g., a responsiveinteraction) to obtain more data from the user should be asked (e.g.,returned to the user as a responsive interaction). These rules 366 maybe rules for evaluating any frames of the current session data for theuser to determine the entities and intents of the previous frames todetermine if enough data is present or the responsive interactions anduser interactions of these previous frames. A rule 366 may, for example,have a trigger that looks for the absence of particular data (e.g., avalue for a particular entity type) and an action that includes a followup question to include as a responsive interaction if the particulardata is not present in the working frame, or an action that populates adefault value for the entity type or other data in the working frame320. For example, certain default values may be: adults: 2, children: 0,departureAirport: airport_closest_to_ip∥‘JFK’, cabin: ‘Economy’, constDEFAULT_DURATION_DAYS=14; DEFAULT_START_DATE=100; // (days from now) orthe like.

The rules 366 may be prioritized such they are evaluated in an order sothe most important or germane follow up question may be asked. Therule(s) 366 to apply may be determined based on the intent (e.g., thecanonical intent) associated with the working frame 320. These rules 366may also determine if default values for entities should be populated inthe working frame 320 if they do not currently exist in the workingframe 320. If the application of the rules 366 results in adetermination that a follow up question should be asked to the user as aresponsive interaction, a follow up question object (e.g., a JSON objectwith a responsive interaction that include the follow up question) maybe generated and included in working frame 320. Such follow up questionsmay include: “Who will you be traveling with?”, “How many adults andchildren will be traveling?”, “When would you like to travel?”, and“What is your budget for this trip?” Additionally, the determination ofa responsive interaction may be based on any exception flags set in theworking frame. For example, if working flags related to specifieddestinations not being served by a partner are set, a particularresponse interaction may be determined and added to the working frame320.

The orchestrator 300 can now process or manipulate the working frame 320using post-processing and response formatting module 316 to create aresponse to the request from the chat widget on the user's system, wherethe response 376 includes a response frame 374 including a set ofcontent objects for presentation to the user through the display portionof the chat system interface and a responsive interaction 376 forpresentation through the chat portion of the chat system interface.

Initially, post-processing and response formatting module 316 may savethe working frame 320 to the session data 358 associated with the usercorresponding to the initially received request. The post-processing andresponse formatting module 316 may then select one or more contentobjects from the working frame for including in the response frame 374for the user. The selection may be based on the entities (includingconcepts) and intents expressed in the current working frame 320, thecurrent session associated with the user, historical sessions associatedwith the user or data returned by a data service 370.

Once the content items and responsive interaction are selected they maybe formatted into the response frame 374 (e.g., as one or more JSONobjects and associated arrays) and included in a response 376 to bereturned to the user. This response frame may thus include a JSON objectcorresponding to cards that are to be displayed in the display portionof the interface (e.g., card display area) and a chat object thatincludes data for rendering in the chat portion of the window. The cardsor response frame 374 may also include map data for calling map serviceto render a map with certain geographical locations or regions in thedisplay portion of the interface (e.g., a map display area).

When the chat system widget at the user's computing device receives thisresponse 376, it may render the response 376 to update the chat driveninterface presented to the user. In particular, the widget may presentthe response interaction (e.g., a natural language or text statementincluded in the response) in the chat portion of the chat driveninterface. Additionally, the chat system widget may render each of thetravel related content objects (e.g., each JSON object associated with atravel related content item) as a card to present in a card display areaof the chat driven interface. Each of these travel related items mayalso have map data (e.g., map data points such as coordinate data suchas latitude or longitude, etc.). Chat system widget may update a mappresented in the map display area of the chat driven interface todisplay these travel related items in the map.

As both the responsive interaction presented in the chat portion of theinterface and the content presented in the display area of the chatdriven interface are based on the context of the human understandableinteractions in the chat potion of the interface, the content presentedin the display portion of the interface reflects the intents or conceptsexpressed in the chat portion of the interface and the chat portion ofthe interface and the display portion of the interface are kept insynchronicity. Moreover, as the user continues to interact with the chatportion of the chat system interface the content displayed in theinterface can also be refined or updated in an ongoing manner by thechat system based on the concepts or intents expressed in a humanunderstandable and comfortable manner through these user interactions.

These chat driven artificial intelligence systems and associatedinterfaces may thus have a wide range of applicability and usefulness,especially in travel related contexts. As noted, the travel industry hasgrown immensely in the previous two decades or so, facilitated in nosmall part by the growth of the Internet. There are therefore a largenumber of web sites or other locations on the Internet that includetravel related content. These sites may be directly travel related, suchas web sites or travel related service providers (e.g., hotel or airlineproviders) or travel related publications (e.g., magazines such as CondeNast or Travel and Leisure), while other web sites may be indirectlyrelated to travel, such as web sites related to activities that may beparticularly associated with certain destinations (e.g., publicationsrelated to scuba diving, sailing, skiing, food, etc.) or blogs or websites related to the travels of individual travelers or certain groups.

Accordingly, as discussed above, in certain embodiments a chat drivenartificial intelligence system may be utilized to provide a chat driveninterface including a chat portion and a display portion within a website provided by a third party publisher or partner (e.g., provided by aparty different than operators of the chat system). In particular, sucha chat driven interface may be presented in a display container (oftenreferred to as an ad container or ad unit) on a publisher's pagealongside travel related content provided by the publisher's web site.The display container may be of the size typically used to display a webbased advertisement such as, for example a 600×300 size container, orthe like (other sizes are imaginable and are fully contemplated herein).The chat driven artificial intelligence system may utilize the contextof the publisher's web site, including for example, the entities(including concepts and destinations) expressed in the content of thepublisher's web page to initialize and drive the chat driven interface,including the content presented in the display portion of the interface.

In particular, in certain embodiments, the content selected forpresentation in the display portion of the chat driven interface may bean advertiser's content (e.g., where that advertiser may be unaffiliatedwith the providers of the publisher's web site). Specifically, the chatdriven system may include an advertiser data store defining concepts,destinations or other entities associated with advertisers or theadvertisers' content. An advertiser (or particular advertiser content)may be selected based on a matching of the concepts, destinations orother entities of the content on the publisher's web page and theconcepts, destinations or other entities associated with the advertiser.Content for the selected advertiser may then be returned and presentedin the display portion of the chat driven interface displayed in the adcontainer on the publisher's web page. The chat system may subsequentlyuse the context of the interactions in the chat potion of the interface(e.g., the dialogue) displayed in the ad unit to drive presentation ofthat advertiser's content in the display portion of the interfacepresented in the ad unit of the publisher's web page such that theadvertiser's content presented in the display portion of the interfacereflects the intents or concepts expressed by the user in the chatportion of the interface.

Referring now to FIG. 4, one embodiment of a computer network topologythat includes an embodiment of a chat driven artificial intelligencesystem (or “chat system”) for presenting a chat driven interface in adisplay container on a publisher's web page is depicted. Chat system 400may be accessed over a network 430 such as the Internet, an intranet, aLocal Area Network (LAN), a Wide Area Network (WAN), a wireless orcellular network, a wireline network or some combination of networks) byusers or applications at computing devices 402 and be implemented ordeployed, for example, on a cloud based platform. To facilitate theseaccesses, a chat interface 410 of the chat system 400 may provide aninterface through which requests may be received and responded to overnetwork 430.

These accesses may be requests from a chat system widget 405 associatedwith a publisher's web page 412. In particular, when a user access theweb site provided by the publisher web server 404 at their device 402using a web browser 406 on their device the publisher's web page 412 maybe provided by the publisher's web server 404. A chat system widget 405may be provided in association with a display container in thepublisher's web page 412.

When the chat system widget 405 is loaded, it may send an initialrequest to the chat system 400 identifying the publisher, thepublisher's web page 412 or other data. The chat system 400 receivesthis request through the chat interface 410 and initializes the chatdriven interface 414 (including the chat portion and the display portionof the interface 414) displayed in the container on the publisher's webpage 412 based on the context of the web page 412. The chat systemwidget 405 may subsequently send requests to the chat interface 410based on a user's interaction with the chat portion of the chat driveninterface 414. These requests may therefore include, the unstructured ornatural language data submitted by the user through the chat portion ofthe chat interface. The chat system 400 receives these requests throughthe chat interface 410 and responds through the chat interface 410. Thechat system widget 405 may receive these responses and process theincluded objects to render or update the chat driven interface 414 inthe publisher's web page 412. As both the responsive interactionpresented in the chat portion of the interface 414 and the contentpresented in the display area of the chat driven interface 414 are basedon the context of the human understandable interactions in the chatpotion of the interface 414, the content presented in the displayportion of the interface 414 reflects the intents or concepts expressedin the chat portion of the interface 414 and the chat portion of theinterface 414 and the display portion of the interface 414 presented inthe context of the publisher's web page 412 are kept in synchronicity.

Specifically, in one embodiment, each publisher 494 where it is desiredto display a chat driven interface from the chat system 400 in a displaycontainer on their web site may be assigned a publisher identifier bythe operators of the chat system 400. The publisher web site index 482may associated the identifier for the publisher with a set of URLs ofthe publisher's web site. Each of the URL's of the publisher's web siteis, in turn, associated with entities (e.g., concepts or destinations,etc.) related to the content accessible at that URL (e.g., thepublisher's web content 408 presented on the web page accessible at thatURL).

To facilitate the collection of data on publishers' web sites, the chatsystem 400 may include site indexer 492. Each publisher's web siteincludes a set of URLs that may be accessed at that publisher's domain.The site indexer 492 may operate periodically (e.g., nightly, weekly,etc.) to index a publisher's web site by accessing the URLs associatedwith the publisher's web site and “scraping” or otherwise obtaining thepublisher's web site content 408 that is presented on each of thepublisher's web pages (e.g., the content accessed at a particular URL).When the content is obtained for a publisher's web page, as this contentusually includes natural language, the site indexer 492 may extractentities (e.g., concepts or destinations, etc.) from the content 408 forthe publisher's web page. The site indexer 492 can employ the dataextractions services 442 or interfaces 444 to take as input the contentscraped from the publisher's web page to extract one or more entities(entity types) from the content and return the extracted entities to thesite indexer 492. These entities may include a set of concepts,including, for example, those related to geography or geographicregions, destinations, time, trip types, activities, amounts, units,names, colors, etc. The data returned from each of the entity extractionservices may be saved into the publisher web site index 482 inassociation with the URL for that publisher's web page.

Advertiser data store 484 may include data associated with advertiserswho desire to present content through a chat driven interface in adisplay unit on a publisher's web page (e.g., where the publisher may becompletely unaffiliated with the advertiser). Each advertiser which itis desired to display a chat drive interface form the chat system 400 ontheir web site may be assigned an advertiser identifier by the operatorsof the chat system 400 and data associated with the advertiser stored inthe advertiser data store 484. This advertiser data 484 may include anidentification of the publishers' web sites on which the advertiser'scontent may be displayed (e.g., because either a publisher desires thatthe content of a particular advertiser not be displayed in associationwith their web site or an advertiser desires that their content not bedisplayed on a particular publisher's site, etc.). The advertiser data484 may also include entities (e.g., concepts or destinations)associated with an advertiser, an identification of particularadvertiser content and entities, or one or more rules associated withthe advertiser, such as a rule defining a business relationship (e.g., acompensation or payment mechanism by which the advertiser pays orotherwise compensates the operators of the chat system 400 inassociation with the display of the advertisers content on a publisher'sweb site, etc.).

Accordingly, when a user at device 402 accesses a publisher's web page412 through browser 406 executing on his device (e.g., by typing a URLassociated with publisher's web site into the browser's address bar,clicking on a link in another web page, etc.), a request for the URLassociated with a particular web page of the publisher's web site issent to the publisher's web server 404 from the user's browser 406. Thepublisher's web server 404 constructs a page 412 to return to the user'sbrowser using the publisher's website content 408. The constructed pagemay include a display container (e.g., defined by a div or iframe tag)associated with a chat system widget 405. The display container may beof the size typically used to display a web based advertisement such as,for example a 600×300 size container, or the like. The chat systemwidget 405 may be incorporated directly into the publisher's web page412 being constructed (e.g., the script code for the chat system widget405 may be include in the page 412 as returned from the web server 404)or a script tag referencing a URL from which the widget 405 may berequested may be included in the web page in association with thedisplay container. The constructed web page 412 is returned from thepublisher's web server 404 to the user's browser 406 where the page 412is loaded in the user's browser and the associated chat system widget405 is executed by the browser 406 or the user's device 402.

When the chat system widget 405 is initially executed (e.g., when thechat system widget is initially loaded by the browser 406 as defined byan “onload” attribute) it may send an initial request to the chat systeminterface 410 of the chat system 400. This request may include the URLof the publisher's web page or other data, including for example, anidentifier for a user or device 402 (e.g. a GUID for the user ordevice), the user's location, IP address, or cookie data associated withthe user or user's browser. This cookie data may include, for example,data from cookies associated with the publisher's web site or cookiesstored and maintained by the chat system 400.

This initial request may be received by the chat interface 410 of thechat system 400. As discussed, the chat interface 410 may be a singleendpoint or may have two or more endpoints (e.g., an init endpoint and achat endpoint), thus when the initial request is received at the chatinterface 410 (e.g., an init endpoint), the chat interface may determinefrom the format of the request (e.g., the set of query parametersassociated with the request) that the request is an initial request froma chat system widget 405 being used on a publisher's web page 412. Ifthe chat interface 410 determines that the request is an initial requestfrom the chat system widget 405 on a publisher's web page 412 it mayprovide that initial request to ad controller 486.

Ad controller 486 may receive this initial request (or the parameters ofthe initial request) including the URL of the publisher web page 412 orthe identifier of the publisher. The ad controller 486 may utilize theURL of the publisher's web page 412 (or the publisher identifier) as anindex into the publisher's web site index 482 to obtain the dataassociated with the URL of the publisher's web page 412, including theentities (e.g., concepts or destinations) associated with thepublisher's web page 412. In one embodiment, the ad controller 486 mayutilize a user (or device) identifier (if one is provided) to access anyhistorical sessions 472 associated with the user. The ad controller 486may determine one or more entities (e.g., concepts or destinations)associated with the user from these historical sessions 472 (e.g., themost recent destinations or concepts returned or associated with theuser in the most recent historical session 472).

Using the entities (e.g., concepts or destinations) associated with thepublisher's web page 412 (as obtained from the publisher web site index482), the ad controller 486 may select an advertiser or associatedadvertiser content, to provide in response to the initial request so theselected advertiser's content can be presented to the user in the chatinterface 414 in the display container on the publisher's web page 412.In particular, the ad controller 486 may include a set of ad selectionrules to select the advertiser or advertiser content from the advertiserdata store 484. These rules may first be applied to determine the set ofadvertisers that are allowed to be displayed on that publisher's website (including the publisher's web page 412). This set of allowableadvertisers can then be scored on relevance by comparing the entitiesassociated with the publisher's web page 412 with the entitiesassociated with the advertisers or the advertisers' content. This scoremay be a similarity score based on the similarity of the concepts,destinations or other entities stored for the publisher's web page 412and for the advertisers or advertisers' content.

The allowable advertisers can then be ranked based on their relevancescores and, in one embodiment, the amount or percentage of compensationassociated with the advertiser (e.g., the amount or percentage theadvertiser will pay operators of the chat system 400 when their contentis displayed on a publisher's web site). The top ranked advertiser canthen be selected. In one embodiment, these ad selection rules may beimplemented as a machine leaning algorithm that may select theadvertiser or content based on, for example, the concepts ordestinations associated with the publisher's web site, the advertisercontent or that the user subsequently selects or interacts with, thedepth of interaction with the chat based interface 414 after suchadvertiser content is presented, or other criteria.

Once the advertiser is selected, advertiser content and a chatinteraction may be determined and returned to the user. In oneembodiment, the advertiser content and chat interaction may bedetermined by ad controller 486 when determining the selectedadvertiser. In particular, the advertiser data store 484 may includecontent for travel related items (e.g., flights, hotels, packages, etc.)associated with the selected advertiser. These travel related items forthe advertiser may also have associated entities (e.g., concepts anddestinations). Thus, the travel related items for the selectedadvertiser may be (e.g., individually) scored based on the based on thesimilarity of the concepts, destinations or other entities stored forthe publisher's web page 412 and for the selected advertiser's travelrelated items. The travel related items may then be ranked based on thedetermined score and one or more travel related items for the advertiserselected. The content for the selected travel related items may be usedto form a response with this content to return to the chat system widget405 through interface 410. This response may include one or more objects(e.g., JSON objects) associated with the selected advertiser content.Additionally, an initial interaction for presentation in the chatportion of the chat system interface may be determined and included inthe response. This initial interaction may be associated with, forexample, a destination, a travel related item, the selected advertiser,or may be a general greeting. Other interactions are possible and arefully contemplated herein.

In such an embodiment, in the event that adequate content for thedetermined advertiser cannot be determined default content and a defaultinitial interaction may be determined. This default content and defaultinteraction may, for example, be stored in the advertiser data store 484in association with the advertiser identifier. The default content may,for example, be a logo or picture with a default interaction of “Ask meabout [publisher's] destinations!” or the like.

The chat system widget 405 may receive the response from the adcontroller 486 and process the included objects to render or update thechat driven interface 414 presented in the container on the publisher'sweb page 412 by presenting the content for the travel related items (ordefault content) in the display portion of the interface 414 and theinitial (or default) interaction in the chat portion of the interface414.

In another embodiment, ad controller 486 may return an advertiseridentifier and a set of entities (e.g., concepts and destinations) tothe chat system widget 405 in response to the initial request from thechat system widget 405. When the chat system widget 405 receives thisresponse, the chat system widget 405 may initiate a subsequent requestto the chat system interface 410 with the advertiser identifier and theset of entities returned from the ad controller 486. The chat systeminterface 410 may receive this subsequent request and determine that therequest should be provided to the chat orchestrator 440. The chatorchestrator 440 may process this request substantially as detailedabove to select content from the identified advertiser and form aresponse with this content and an interaction to return to the chatsystem widget 405 through interface 410. These responses may include oneor more objects associated with the selected advertiser content. Thechat system widget 405 may receive this response and process theincluded objects to render or update the chat driven interface 414presented in the container on the publisher's web page 412 by presentingthe content for the advertiser's travel related items in the displayportion of the interface 414 and the interaction in the chat portion ofthe interface 414.

In the manner described above, embodiments may utilize the entities(e.g., concepts and destinations) of the publisher's web site content408 to initialize a chat driven interface 414 presented in a displaycontainer included in that publisher's web page 412, includinginitialization with contextual advertiser content. Moreover, once thechat driven interface 414 presented in the display container on thepublisher web page 412 is initialized (e.g., initial content or aninitial interaction for the chat portion is presented in the interface414) the chat system widget 405 may interact with the chat system 400substantially as described above such that the chat orchestrator 440 mayprocess these subsequent requests to return the selected advertiserscontent and responsive interactions to the user through the chat driveninterface 414 presented in the display container.

As in many cases, there may be established and extensive networks forthe distribution of content through ad containers on publisher's webpages, and publishers may already be extensively familiar and integratedwith such ad distribution networks, in certain embodiments these addistribution networks may be leveraged to present chat driven interfacesin display container's on a publisher's web page. In particular, such achat driven interface may be presented in a display container (oftenreferred to as an ad container or ad unit) on a publisher's pagealongside travel related content provided by the publisher's web site.The display container may be of the size typically used to display a webbased advertisement such as, for example a 600×300 size container, orthe like (other sizes are imaginable and are fully contemplated herein).The chat driven artificial intelligence system may utilize the contextof the publisher's web site, including for example, the entities(including concepts and destinations) expressed in the content of thepublisher's web page to initialize and drive the chat driven interface,including the content presented in the display portion of the interface.

Moving then to FIG. 5, one embodiment of a computer network topologythat includes an embodiment of a chat system for presenting a chatdriven interface in a display container on a publisher's web page usingan ad distribution network is depicted. Again, chat system 500 may beaccessed over a network 530 such as the by users or applications atcomputing devices 502 where this access is facilitated through chatinterface 510. Here, chat system 500 may provide a publisher interface503 that offers a set of interface where a publisher 594 or userassociated therewith may define the “look and feel” or associatedcontent or initialization of the chat driven interface that will appearin the publisher's web page 512. In this manner, a publisher 594 maydefine multiple different chat driven interfaces for different purposesor “campaigns”. Additionally, in one embodiment, a publisher 594 mayutilize the publisher interface 503 to specify one or more concepts orentities for use with a particular chat driven interface.

Based on the publisher's definition for the chat driven interface atemplate for each particular definition can be stored as templates 578associated with the publisher 594, an identifier for the publisher 594,or the identifier for that particular configuration for the chat driveninterface. Moreover, a corresponding chat system widget 505 for thepublisher's definition may be created. This chat system widget 505 maybe configured according to the definition by configuring the chat systemwidget 505 to make a call (e.g., a request to the/init endpoint of thechat interface 510) for the template 578 associated with thatdefinition. Such a configuration may be, for example, including a tag orother code in the widget 505 that references the associated template 578(e.g., includes an identifier for the associated template) and willcause a call to be made that includes the reference to the template. Incases where the publisher's definition includes concepts or entities,the associated chat system widget 505 may also be configured accordingto those concepts or entities by including these concepts or entities inthe a tag or other code in the widget 505 that will cause a call to bemade. Thus, such a call to the chat interface 510 may also include suchconcepts or entities. These chat system widgets 505 may then be storedin publisher widget store 582.

At some point then, these chat system widgets 505 may be deployed withinan ad distribution network provided by an ad distribution networkprovider 599 such as DoubleClick by Google or the like. Thus, the chatsystem widgets 505 may be deployed at the ad distribution networkprovider 599 and may be served to a user's web browser 506 when a useraccess a publisher's web page 512. For example, a publisher's web page512 may include a display container that may reference the addistribution network provider 599 and a particular campaign or chatsystem widget 505 (e.g., may include an identifier associated with thecampaign or widget 505).

Accordingly, when a user access the web site provided by the publisherweb server 504 at their device 502 using the web browser 506 on theirdevice the publisher's web page 512 may be provided by the publisher'sweb server 504. As the display container of the publisher's page 512references the ad distribution network provider 599, a request may besent to the ad distribution network provider 599. The request mayinclude reference the publisher's campaign of chat system widget 505. Inresponse to this call, the ad distribution network provider 599 mayprovide the referenced chat system widget 505 to the requesting browser506 where it is loaded in the display container of the publisher's webpage 516.

As has been discussed the chat system widget 505 is configured to whenthe chat system widget 505 is loaded in the display container (e.g., anexecuted by the browser 506), it may send an initial request to the chatsystem 500 identifying the URL of the publisher's web page and atemplate 578 associated with the chat system widget 505. If the chatsystem widget 505 was configured with concepts or entities these mayalso be provided in association with the request. Additionally, therequest may include one or more tags or other metadata provided by thead distribution network provider 599 in association with the publisher'sweb page 512.

To illustrate in more detail, in many cases ad distribution networkproviders 599 or publishers' 594 may provide metadata in associationwith the web page 512 or display container (e.g., frame) on thepublisher's page 512 in which the content is to be presented. Thismetadata may, for example, define the web context or other dataregarding the web page 512 in which the display container exists. Thismay allow, for example, tracking or analytics to be performed inassociation with the publisher's web page 512 or advertisementspresented within that display container in association with thepublisher's web page 512. Examples of such tags include Urchin TrackingModule (UTM) parameters or the like.

In one embodiment, chat system widget 505 when composing an initialrequest to the chat system 500, may access such metadata if it ispresent or available with respect to the publisher's web page andinclude such metadata as parameters to this initial request. Forexample, the chat system widget 505 may access any utm_term parametersincluded in association with the publisher's web page 512 and includethese terms as parameters to the request.

The chat system 500 may utilize the identifier of the template 578 toaccess the template 578 associated with the chat system widget andreturn the template 578 along with a response frame including content todisplay in the display portion of the chat driven interface 514 or achat interaction to present in the chat portion of the interface 514. Inparticular, the chat system 500 may receive the request and determineany concepts or entities associated with the request. These concepts andentities may be determined from the request and may have originated, forexample, from the configuration of the chat system widget 505 or mayhave been included in the request based on metadata as determined fromthe publisher's web page 512 or display container included therein.

In one embodiment, concepts or entities for the publisher's web page 512may be determined in real-time at the point an initial request isreceived from the chat system widget 505. Specifically, such a requestmay include the URL of the publisher's web page or web site. When therequest is received at the chat interface 510, the URL of thepublisher's web page 512 or web site may be provided to site indexer592. The site indexer 592 may operate in real-time to index apublisher's web site by accessing the URL associated with thepublisher's web site and “scraping” or otherwise obtaining thepublisher's web site content that is presented on the publisher's webpage 512 (e.g., the content accessed at the URL). When the content isobtained for a publisher's web page the site indexer 592 may extractentities (e.g., concepts or entities, etc.) from the content for thepublisher's web page 512. These concepts or entities may be returned bysite indexer 592 to the chat interface 510 for inclusion in the requestto the chat orchestrator 540.

Additionally, in one embodiment, a site index 594 may be created by thesite indexer 592 which maps URLs of publisher web pages 512 toassociated extracted concepts or entities. Accordingly, when the requestis received at the chat interface 510, the URL of the publisher's webpage 512 or web site may be provided to site indexer 592. Here, the siteindexer 592 may access site index 594 to determine if the web page 512associated with the URL has already been indexed. If so, the concepts orentities may be determined from the site index 594 based on the URL andreturned to the chat interface 510 for inclusion in the request to thechat orchestrator 540. If the web page 512 associated with the URL forthe web page 512 has not been indexed, it may be indexed in real-time asdiscussed.

Alternatively, in one embodiment, the site indexer 592 may return anerror or an indication that no concepts or entities have been found tothe chat interface 510. The site indexer 592 can then access the webpage 512 associated with the URL can then be indexed asynchronously tothe received request at a later point and added to the site index 592such that subsequent requests for that URL can be satisfied utilizingthe site index 594.

The request received from the chat widget 505 at the chat interface 510,including any included or determined concepts and entities may be passedto orchestrator 510. The chat orchestrator 540 may process this requestsubstantially as detailed above to select content and form a responsewith the template 578, the selected content and an interaction to returnto the chat system widget 505 through interface 510. These responses mayinclude one or more objects associated with the selected content. As maybe noted, this may be content particular to an advertiser as discussedabove, may be content associated with a data service of the chat system500 or may be some combination. The chat system widget 505 may receivethis response and process the included objects to render or update thechat driven interface 514 presented in the container on the publisher'sweb page 512 by presenting the content in the display portion of theinterface 414 and the interaction in the chat portion of the interface514.

In the manner described above, embodiments may utilize the entities(e.g., concepts and destinations) as determined from the publisher'sconfiguration or from metadata associated with the web page by thepublisher 594 or ad distribution network provider 599 to initialize achat driven interface 514 presented in a display container included inthat publisher's web page 512, including initialization with contextualcontent. Moreover, once the chat driven interface 514 presented in thedisplay container on the publisher web page 512 is initialized (e.g.,initial content or an initial interaction for the chat portion ispresented in the interface 514) the chat system widget 505 may interactwith the chat system 500 substantially as described above such that thechat orchestrator 540 may process these subsequent requests to returnthe selected content and responsive interactions to the user through thechat driven interface 514 presented in the display container.

As will be recalled from the above discussion, embodiments of the chatdriven artificial intelligence system may utilize one or more dataservices to determine the content to provide to a user. Specifically, incertain embodiments, the orchestrator may provide extracted concepts orintents to one or more data services. These data services may provideone or more travel related items (e.g., destinations, regions, hotels,neighborhoods, etc.) in response, along with content related to thosetravel related items. In embodiments, the data services include a set ofservices that may take as input a concept and search concept data torank a set of travel related items and return the content for the topranked set of travel related items.

As the goal of embodiments of chat driven artificial intelligencesystems is to select and provide accurate content reflective of theconcepts or destinations expressed by the user it may therefore bedesirable in many instances to premise the search for items and relatedcontent on a knowledge base that includes concepts (referred to hereinas a concept network). This is the case at least because the conceptsthat may be used to search for items and related content are notnecessarily expressly provided by the user. Instead, in many cases theconcepts used for this searching may be derived from the naturallanguage statements of the user (e.g., as provided in a chat interface)or other sources.

Accordingly, embodiments of the data services of the chat drivenartificial intelligence system may create, maintain or utilize conceptnetworks. These concept networks may include a graph comprised of nodesand edges, where the nodes may include concept nodes and the edges mayinclude relationships between these concepts nodes (e.g., concepts) andother nodes, including other concept nodes or nodes associated withvarious items. In a travel related context, for example, these nodes mayrepresent items such as hotels, destinations, flights, attractions,activities, etc. The relationships indicated by the edges of the conceptnetwork may be assigned weights or scores indicating a degree ofaffinity between the nodes related by a relationship, including, forexample, the degree of affinity between a concept represented by aconcept node and an item node to which that concept node is related.Additionally, the relationships may be directional, such that they maybe traversed only in a single direction, or have different weightingsdepending on the direction in which the relationship is traversed or thenodes related.

Embodiments of such a concept network can thus be searched (ornavigated) to determine items (and related content) associated with oneor more concepts. Moreover, the degree of affinity between the conceptsand items may be determined using the weights of the relationships inthe concept network. In particular, concept networks including nodesrepresenting travel related items such as hotels or destinations may beused in travel related systems to search for and provide these travelrelated items (and related content) associated with one or moreconcepts. In this manner, the travel related content related to theconcepts expressed by, or determined from, a user may be utilized toprovide insightful items and content to the user. It should be notedhere that while concept networks and the creation, updating and use ofsuch concept networks are particularly germane to the context of travelrelated artificial intelligence systems and that is the context in whichembodiments are illustrated, after a review of this disclosure those ofordinary skill will understand that embodiments of concept networks andtheir use may be applicable to, and utilized in, a wide variety of othercontext, all of which are fully contemplated herein.

Specifically, a concept network may be thought of as a graph comprisinga number of interrelated nodes. These nodes may include nodes that mayhave labels defining the type of the node (e.g., the type of “thing” orentity that the node represents) and properties that define theattributes or data of that node. In a travel related context then, sucha concept network may associate nodes representing travel related itemswith concepts. Moreover, the travel related items may also be associatedwith content for those travel related items. The content can be storedas a property of the node representing a travel related item or as partof a distinct node related to the node representing the travel relateditem. For example, the labels of the nodes of a concept network mayinclude CONCEPT, IMAGE, USER, HOTEL, FLIGHT, DESCRIPTION, EVIDENCE,IMAGE, DESTINATION, ATTRACTION, or the like. Properties of a node mayinclude, NAME:[NAME], SOURCE:[SOURCE], SYNONYMS:[LIST OF SYNONYMS], orthe like.

Thus, the concepts represented in a concept network may include almostany concepts desired. These concepts have concept nodes (e.g., a nodewith the label “CONCEPT”) with one or more properties defining theconcept (e.g., names such as “pool”, “spa”, “kid”, “brisket”, “$”, “$$,etc.). In certain embodiments, the names for concepts may be uni-grams,bi-grams or tri-grams (e.g., concepts that may be expressed in one, twoor three word or symbol groups).

The nodes of the concept network may be interrelated using relationshipsthat form the edges of the graph. A relationship may connect two nodesin a directional manner. These relationships may also have a label thatdefines the type of relationship and properties that define theattributes or data of that relationship. These properties may include anidentification of the nodes related by the relationship, anidentification of the directionality of the relationship or a weight ordegree of affinity for the relationship between the two nodes. Forexample, the labels of the relationships of a concept network mayinclude “isRelatedto” (e.g., for relationships between concept nodes),“hasConcept”, “hasDescription”, “hasEvidence”, “hasAttraction”, or thelike.

It may be useful here to an understanding of concept networks toillustrate an example of such a concept network. It will be noted thatwhile concept networks as described herein have been described asgraphs, these concept networks may be arranged or stored in any suitablemanner (e.g., data structures or the like) that captures such conceptsand relationships, including in a graph database such as Neo4j, in arelational database such as Postgres, in another type of database, in aflat file structure or otherwise.

Referring then to FIG. 6A, a graphical depiction of a portion of anexample concept network 5000 is depicted. Here, nodes are represented bycircles and relationships are represented by the directional arrowsbetween the nodes. Such a concept network 5000 may represent concepts,their association and the degree of affinity between entitiesrepresented by the nodes. Thus, for example, the concept node 5100 a hasa label “CONCEPT” indicating it is a concept node and a name “Ocean”indicating the concept node represents the concept of an ocean.Similarly, concept node 5100 b has a label “CONCEPT” indicating it is aconcept node and a name “Surfing” indicating the concept node representsthe concept of surfing. These nodes of the graph are joined by edgesformed by directed relationships 5102 a, 5102 b representing that theconcept of Ocean (represented by node 5100 a) is related to (representedby the labeled “isRelatedto” relationship 5102 a) the concept of Surfing(represented by node 5100 b), and the concept of Surfing (represented bynode 5100 b) is related to (represented by the labeled “isRelatedto”relationship 5102 b) the concept of Ocean (represented by node 5100 a).

Note that each relationship 5102 may have a different weight (even inthe case where the relationship 5102 is the same type of relationshipbetween the same nodes). The ability to assign different weights todirectional relationships (even the relationships of the same typebetween the same two nodes) may allow a concept network to reflect, forexample, that there is a hierarchy to some concepts. For example, in theinstant example, the concept of surfing may be strongly tied to theconcept of ocean, but the concept of ocean may be much broader than theconcept of surfing. It will also be noted that in certain embodiments,these weights may be negative indicating, for example, a strong degreeof dissimilarity.

These concept networks not only represent that concepts are related toone another, but additionally may also represent that entities(including real-world entities such as hotels, destinations, flights,etc.) are associated with certain concepts and the degree of affinitybetween these concepts and entities. As an example in concept network5000, the concept node 5100 c has a label “CONCEPT” indicating it is aconcept node and a name “Spa” indicating the concept node represents theconcept of a spa. Similarly, node 5100 d has a label “HOTEL” indicatingit is a hotel node and a name “LagoVista” indicating the hotel noderepresents the LagoVista hotel. These nodes of the graph are joined byedges formed by directed relationships 5102 c, 5102 d representing thatthe concept of Spa (represented by node 5100 c) is related to(represented by the labeled “isRelatedto” relationship 5102 c) theLagoVista hotel (represented by node 5100 d), and the LagoVista hotel(represented by node 5100 d) is related to (represented by the labeled“hasConcept” relationship 5102 d) the concept of spa (represented bynode 5100 c).

A concept network may also represent relations between entities. Inconcept network 5000 for example, node 5100 e has a label “Destination”indicating it is a destination node and a name “Austin” indicating thedestination node represents the destination Austin. The Austindestination node 5100 e is joined to the LagoVista hotel node 5100 d byedges formed by directed relationships 5102 e, 5102 f representing thatthe LagoVista hotel (represented by node 5100 d) is related to(represented by the labeled “isLocatedin” relationship 5102 e) thedestination of Austin (represented by node 5100 e), and the destinationof Austin (represented by node 5100 e) is related to (represented by thelabeled “hasHotel” relationship 5102 f) the LagoVista hotel (representedby node 5100 d).

As can be seen from the example illustrated, a concept network canrepresent a wide variety of entities and concepts, and the relationshipsbetween them, in a graph like manner. The consanguinity (if any) of anytwo nodes in the concept network can thus be determined according to apath between the two nodes and the degrees of affinity associated withthe relationships of that path. It will be noted that these conceptnetworks may be stored in any suitable format and according to anysuitable storage, including, for example, a graph store such as Neo4j orOrientDB, a triple store, a relational database such as Postgres, etc.Access and queries to such a concept network may thus be accomplishedusing an associated access or query language (e.g., such as Cypher inthe case where the Neo4j graph store is utilized or Elasticsearch orPostgresql in the case of Postgres, etc.).

One particular embodiment of a concept network may be referred to as atrie. A trie can be thought of as a search tree data structure used tostore a dynamic set or associative array of concepts. Specifically, incertain embodiments, the trie may be a concept network including ahierarchically set of concept nodes arrayed as a tree structure. Thus,each node at each level of the tree may be a concept node representing aconcept that is a sub-concept of a parent concept node (as representedby the presence of an edge denoting that the node is a sub-node of theparent concept node) and where each node may have one or more sub-nodeseach representing a sub-concept of that node. In this manner, a trie canbe traversed from any node to any other node. Weights or scores (usedinterchangeably herein) can be generated between any two concepts in atrie. In one embodiment, such a weight may be based on the number ofedges (or distance) between two concepts, the number of levels of thetrie that must be traversed, the direction of traversal (e.g., either upthe trie or down the trie) to navigate between the two nodesrepresenting those concepts or other traversal or traversal distancebased criteria.

FIG. 6C is a graphical depiction of a portion of an example conceptnetwork 6000 arranged as a trie. Again, nodes are represented by circlesand relationships are represented by the directional arrows between thenodes. Such a concept network 6000 may represent concepts whilerelationships in trie represent a hierarchy of the concepts in the trie.Thus, for example, the concept node 6100 a has a label “CONCEPT”indicating it is a concept node and a name “Landmarks” indicating theconcept node represents the concept of a landmarks. Similarly, conceptnode 6100 b has a label “CONCEPT” indicating it is a concept node and aname “Historical” indicating the concept node represents the concept ofHistorical. These nodes of the graph are joined by edge formed byrelationship 6102 a, representing that the concept of Historical as(represented by node 6100 b) is a sub-node or sub-concept of the conceptof Landmarks (represented by node 6100 a). Concept node 6100 c has alabel “CONCEPT” indicating it is a concept node and a name “Science”indicating the concept node represents the concept of Science. Conceptnode 6100 c is joined by an edge formed by relationship 6102 b,representing that the concept of Science (represented by node 6100 c) isa sub-node or sub-concept of the concept of Landmarks (represented bynode 6100 a). In the same manner, it can be seen from the depictedportion of a trie that the concept of Architecture is a sub-node orsub-concept of both Historical and Landmark.

Now with an idea of concept networks in mind, attention is directed toartificial intelligence systems and associated data services that mayemploy these concept networks in the context of travel. Specifically, incertain embodiments, searching data services utilizing a concept networkmay be provided whereby one or more concepts may be provided to thesearching data service through an interface. These searching dataservices may construct a search for a particular type of travel orreal-estate related item (e.g., a destination or hotel) based on theconcepts received and return an ordered list of items (and relatedcontent) based on the degree of affinity between the received conceptsand the items as represented in the concept network.

Utilizing embodiments of a concept network, such searches may accuratelyreturn real-world travel or real-estate related items and relatedcontent associated with concepts (such as images or the like) to a user.Moreover, these concept networks may be automatically created or updatedwithout user involvement based on a wide variety of criteria tocontinually improve the accuracy or performance of the data servicesperforming such searches. Additionally, embodiments may employ certaintypes of relationships, types of concept networks, storage of theconcept network, or divisions of the concept network, to achieve greaterspeed or reduce the processing or memory required in the storage,searching or other processing accomplished in association with suchcreating, updating, searching or storing concept networks.

Again, it should be noted here that while embodiments of such searchingtechniques, methods and systems have been described herein with respectto the context of, and use in, interaction driven artificialintelligence system, embodiments of such search techniques, methods andsystem, including those used for searching or recommendations ofdestinations or hotels and associated content such as images may beutilized usefully in a variety of other contexts where the searching orrecommendation of destinations, hotels or certain content (e.g., such asimages) may be desired and the embodiments and context of thedescription of various embodiments herein is provided without loss ofgenerality.

FIG. 7 depicts one embodiment of a travel related artificialintelligence system 600 employing concept networks in association with adata service that may be utilized to search using these conceptnetworks. Certain depicted components operate substantially similarly tothose discussed above and will not be discussed in association with FIG.7. In this embodiment, different concept networks for different entitiesmay be created or stored separately in order to, for example, increasethe performance of the searching or other interactions with the conceptnetworks. These concept networks may be used in association with otherdata by the data services 650 to return travel related items andassociated content in response to requests received at those dataservices 650. These data services 650 may, for example, be implementedas microservices or web services that are accessed using an API, such asa RESTful interface or the like.

In particular, a destination concept network 612 may be a conceptnetwork that includes nodes representing destinations, other entitiesrelated to those destinations or concepts related to those destinationsand entities. Such a destination concept network 612 may be stored, forexample, in a graph database such as Neo4j, in a relational databasesuch as Postgres, in another type of database, in a flat file structureor otherwise. In certain embodiments, this destination concept network612 may not include nodes representing certain entities such as hotelsor other accommodations (collectively referred to herein as hotels).

While destination concept network 612 may include nodes corresponding todestinations incorporated into the concept network 612, in oneembodiment, the destinations 692 may be stored separately from theconcept network 612 (e.g., the concept network 612 may not include nodesrepresenting destinations) or a separate listing of destinations 692 maybe maintained in addition to the nodes in the concept network 612representing destinations. Moreover, there may be an association ofdestinations with their (geographical) locations. This association maycomprise, for example, an association between a destination and one ormore GeoJSON objects for that destination (or may be a JSON object forthe destination including such a GeoJSON) in the destinations 692. Thelisting of destinations 692 may be maintained in a graph database suchas Neo4j, in a relational database such as Postgres, in another type ofdatabase, in a flat file structure or otherwise.

Content associated with those destinations may also be stored inassociation with those destinations. For example, in one embodimentimages associated with those destinations may be stored in associationwith those destinations. One or more concepts may be associated withthese images. Such concepts may, for example, correspond to the conceptsused by these system where such concepts may be determined by a thirdparty data service or from the context of the source from which theimage was obtained. In some embodiments, a score may be associated witheach of the concepts associated with the image. These scores may takethe form, for example, of an image concept vector including a scoreassociated with the concepts (e.g., the entire set or a subset of)utilized by the system.

In one embodiment, region data 618 may be stored separately and includedata on various geographical regions. Specifically, region data 618 mayinclude one or more GeoJSON objects that define or bound an associatedregion. Thus, region data 618 may be accessed to determine the metes orbounds of a particular region or destination. This region data 618 maybe stored in a relational database such as Postgres.

Similarly, hotel concept network 614 may be a concept network thatincludes nodes representing hotels and other entities related to thosehotels and concepts related to those hotels and entities. Such a hotelconcept network 614 may be stored maintained in a graph database such asNeo4j, in a relational database such as Postgres, in another type ofdatabase, in a flat file structure or otherwise. In certain embodiments,this hotel concept network 614 may not include nodes representingcertain entities such as destinations.

Again, while hotel concept network 612 may include nodes correspondingto destinations incorporated into the concept network 612, in oneembodiment, the hotel 694 may be stored separately from the hotelconcept network 614 (e.g., the hotel concept network 614 may not includenodes representing hotels) or a separate listing of hotels 694 may bemaintained in addition to the nodes in the hotel concept network 614representing hotels. Moreover, there may be an association of hotelswith their (geographical) locations. This association may comprise, forexample, an association between a destination and one or more GeoJSONobjects for that hotel (or may be a JSON object for the hotel includingsuch a GeoJSON) in the hotels 694. The listing of hotels 694 may bemaintained in a graph database such as Neo4j, in a relational databasesuch as Postgres, in another type of database, in a flat file structureor otherwise.

In one embodiment, this hotel destination data 616 may be storedseparately from the listing of hotels 694. In such an embodiment, hoteldestination data 616 may be an association of hotels with theirlocations. This association may comprise, for example, an associationbetween a hotel and one or more GeoJSON objects for that hotel (or maybe a JSON object for the hotel including such a GeoJSON). Hoteldestination data 616 may, for example, be stored maintained in a graphdatabase such as Neo4j, in a relational database such as Postgres, inanother type of database, in a flat file structure or otherwise.

As with destinations, content associated with those hotels may also bestored in association with those hotels. For example, in one embodimentimages associated with those hotels may be stored in association withthose hotels. One or more concepts may be associated with these images.Such concepts may, for example, correspond to the concepts used by thesesystem where such concepts may be determined by a third party dataservice or from the context of the source from which the image wasobtained. In some embodiments, a score for that image may be associatedwith each of the concepts associated with the image. These scores maytake the form, for example, of an image concept vector including a scoreassociated with the concepts (e.g., the entire set or a subset of)utilized by the system.

It will be understood that while certain and various data and structuresdescribed have been depicted separately or as being stored together,other embodiments may combine or separate such data or structuresaccording to the context or implementation of such embodiments. Forexample, in one embodiment, there may a single concept network thatincludes all concepts utilized by the chat system 600 and separatedestination listings and hotel listings may be maintained (or may becombined into a single listing). In this manner, there may be a set ofcanonical or complete set of concepts that may be utilized by the chatsystem 600 for both destinations and hotels.

As discussed above, a data service 650 in the context of a chat drivenartificial intelligence system may receive a request comprising one ormore concepts and return one or more entities or content for one or moreentities related to the concepts. One data service 650 may be a hoteldata service 620. This hotel data service 620 includes an interface 622for receiving and responding to requests and a search module 624. Thehotel data service 620 receives requests through the interface 622 andutilizes the search module 624 in association with hotel concept network612 to determine hotels responsive to the concepts of the receivedrequest and return content associated with those hotels through theinterface 622. The request may include one or more concepts andoptionally one or more regions. The response to this request may includeone or more ordered JSON objects, the JSON objects including content fora set of ordered or ranked hotels.

In one embodiment, if a region is included in the request to the hoteldata service 620 the search module 624 may first access the region data618 to determine the metes and bounds of the regions. The search module624 may query the region data 618 with the region specified in therequest and obtain the GeoJSON associated with the region. Using thespecifications for the region (e.g., the boundaries as specified in theGeoJSON for the region) the search module 624 may then query the hoteldestination data 616 to determine all the hotels located within thatregion.

Once the set of hotels within the region are determined, each of thosehotels can be scored and ranked based on the hotel concept network 614.The determination of a score between the concepts of the request and ahotel can be based on both the number of “hops” or nodes along the pathtraversed between the node representing the concept and the noderepresenting the hotel, along with the degree of affinity (weight) ofeach relationship in the path.

Specifically, to score a hotel each of the concepts of the request isevaluated to determine the shortest path from the node representing thehotel to the node representing that concept. Each of the paths that isshortest (e.g., the fewest number of “hops” or relationships between thetwo nodes) may be scored by multiplying the weights of each relationshipin that path. The shortest path with the highest score may be selectedas the score for that concept with respect to that hotel. In oneembodiment, there may be a maximum “depth” or number of hops that a pathmay have (e.g., 2, 3 or 4 hops), such that if a path includes more thanthis maximum number of hops it will not be considered as a possibleshortest path or will otherwise not be utilized. If there is no shortestpath between the concept (node) and the hotel (node), either becausethere is no path at all or because there is no path with less than amaximum depth, the score for that concept and that hotel may be assigneda zero value. Once a score for each concept is determined for the hotel,the total score for that hotel may be determined based each of theindividual scores for each of the concepts of the request. This hotelscore may be determined for example by summing the scores for each ofthe individual concepts, multiplying the scores (e.g., excluding anyzero scores), averaging the scores for the concepts or otherwise. Inthis manner, scores for each of the hotels in the region may bedetermined using the hotel concept network 614 and the concepts,relationships and hotels represented therein.

A number of hotels with the highest score (e.g., the top 5, top 10,etc.) may then be selected and the content for these hotels determined.As discussed, this content for a hotel may be included in the node ofthe hotel concept network 614 representing that hotel or may be includedin other nodes related to the node for the hotel. The content for thehotel may be included in an object for that hotel (e.g., a JSON objector the like) and the objects for each of the top ranked hotels may bereturned by the hotel data service 620 in response to the initiallyreceived request. In one embodiment, images associated with a determinedhotel that are likewise associated with the concepts of the receivedrequest may be determined and these images returned as at least aportion of the content associated with those hotels.

In one embodiment, the determination of the images to include in thecontent for a selected hotel may be based on one or more scores forconcepts (e.g., the concepts received by the hotel data service 620)that are associated with the set of images for the determined hotel. Forexample, the set of images associated with the hotel may be accessed,and scores associated with the one or more concepts received in therequest to the hotel data service 620 may be determined for each of theset of images based on the scores for those concepts associated witheach particular image of the set. The scores for these concepts may beincluded, for example, in an image concept vector for each image. One ormore images can then be selected from the set of images for inclusion inthe content for the hotel based on the score determined for the imagesutilizing the scores for the received concepts associated with eachimage.

Another data service that may be utilized by embodiments of a travelrelated artificial intelligence system may be a destination data service640. This destination data service 640 includes an interface 642 forreceiving and responding to requests and a search module 644. Thedestination data service 640 receives requests through the interface 642and utilizes the search module 644 in association with destinationconcept network 612 to determine destinations responsive to the conceptsof the received request and return content associated with thosedestinations through the interface 642. The determination of a degree ofaffinity between the concepts of the request and a destination can bebased on both the number of “hops” or intervening nodes along the pathtraversed between the node representing the concept and the noderepresenting the destination, along with the weights of eachrelationship in the path. The destination data service 640 may also thesearch module 644 in association with destination concept network 612 todetermine images associated with determined destinations that arelikewise associated with the concepts of the received request and returnthese images as at least a portion of the content associated with thosedestinations through the interface 642.

Specifically, each of the destinations of the destination conceptnetwork 612 may be scored. To score a destination each of the conceptsof the request is evaluated to determine the shortest path from the noderepresenting the destination to the node representing that concept. Eachof the paths that is shortest (e.g., the fewest number of “hops” orrelationships between the two nodes) may be scored by multiplying theweights of each relationship in that path. The shortest path with thehighest score may be selected as the score for that concept with respectto that destination. In one embodiment, there may be a maximum “depth”or number of hops that a path may have (e.g., 2, 3 or 4 hops), such thatif a path includes more than this maximum number of hops it will not beconsidered as a possible shortest path or will otherwise not beutilized. If there is no shortest path between the concept (node) andthe destination (node), either because there is no path at all orbecause there is no path with less than a maximum depth, the score forthat concept and that destination may be assigned a zero value. Once ascore for each concept is determined for the destination, the totalscore for that destination may be determined based each of theindividual scores for each of the concepts. This destination score maybe determined for example by summing the scores for each of theindividual concepts, multiplying the scores (e.g., excluding any zeroscores), averaging the scores for the concepts or otherwise. In thismanner, scores for each of the destinations may be determined using thedestination concept network 612 and the concepts, relationships anddestinations represented therein.

A number of destinations with the highest score (e.g., the top 5, top10, etc.) may then be selected and the content for these destinationsdetermined. As discussed, this content for a destination may be includedin the node of the destination concept network 612 representing thatdestination or may be included in other nodes related to the node forthe destination. The content for the destination may be included in anobject for that destination (e.g., a JSON object or the like) and theobjects for each of the top ranked destination may be returned by thedestination data service 640 in response to the initially receivedrequest. In one embodiment, images associated with a determineddestinations that are likewise associated with the concepts of thereceived request may be determined and these images returned as at leasta portion of the content associated with those destinations.

In one embodiment, the determination of the images to include in thecontent for a selected destination may be based on one or more scoresfor concepts (e.g., the concepts received by the destination dataservice 640) that are associated with the set of images for thedetermined destination. For example, the set of images associated withthe destination may be accessed, and scores associated with the one ormore concepts received in the request to the destination data service640 may be determined for each of the set of images based on the scoresfor those concepts associated with each particular image of the set. Thescores for these concepts may be included, for example, in an imageconcept vector for each image. One or more images can then be selectedfrom the set of images for inclusion in the content for the destinationbased on the score determined for the images utilizing the scores forthe received concepts associated with each image.

As may be imagined, the complexity of various concept networks asdiscussed herein may be immense, with tens or hundreds of thousands (ormore) of different nodes in each. The traversal of these conceptnetworks to determine the shortest path(s) when determining a set of topranked hotels or destinations may therefore be computationally expensive(e.g., from a processor or memory perspective) and relatively timeconsuming. It may thus be desired to configure the various conceptnetworks such that the navigation of these concept networks may beimproved, reducing the use of computational resources need to accomplishsuch navigation and speeding the navigation of these concept networks(and thus commensurately speeding the search and reducing response timesfor systems and processes that must navigate these concept networks).

Accordingly, in one embodiment, concept networks 612, 614 may include“metaRelated” relationships between two or more nodes. A metaRelatedrelationship indicates that there is a path comprising multiplerelationships (e.g., longer than a single hop) between the two nodesrelated by the metaRelated relationship. Moreover, the metaRelationshipmay have an associated weight reflective of the weights of therelationships in the path between the two nodes represented by themetaRelated relationship.

It may be useful to briefly illustrate the use of metaRelatedrelationships with the reference to the previously provided example of aconcept network. Referring then to FIG. 6B, the example concept network5000 is depicted along with metaRelationships that have been added. Asdiscussed, concept node 5100 b has a label “CONCEPT” indicating it is aconcept node and a name “Surfing” indicating the concept node representsthe concept of surfing, concept node 5100 a has a label “CONCEPT”indicating it is a concept node and a name “Ocean” indicating theconcept node represents the concept of an ocean and concept node 5100 fhas a label “CONCEPT” indicating it is a concept node and a name “Water”indicating the concept node represents the concept of water.

These nodes of the graph are joined by edges formed by directedrelationships 5102 b, 5102 i representing that the concept of Surfing(represented by node 5100 b) is related to (represented by the labeled“isRelatedto” relationship 5102 b) the concept of Ocean (represented bynode 5100 a) with a weight of 0.9, and the concept of Ocean (representedby node 5100 a) is related to (represented by the labeled “isRelatedto”relationship 5102 i) the concept of Water (represented by node 5100 f)with a weight of 0.8. The metaRelated relationship 5102 g representsthis shortest path from the Surfing concept node 5100 b to the Waterconcept node 5100 f along the path defined by relationship 5102 b, Oceannode 5100 a and relationship 5102 i. Additionally, metaRelatedrelationship 5102 g may be assigned a weight of 0.72 reflecting theweights of the relationships 5102 b, 5102 i in that shortest path (e.g.,the product of the 0.9 weight of relationship 5102 b and the 0.8 weightof relationship 5102 i).

Similarly, the metaRelated relationship 5102 h represents the shortestpath from the Surfing concept node 5100 b to the Pool concept node 5100g along the path defined by relationship 5102 b, Ocean node 5100 a,relationship 5102 i, Water node 5100 f and relationship 5102 j.Additionally, metaRelated relationship 5102 h may be assigned a weightof 0.575 reflecting the weights of the relationships 5102 b, 5102 i and5102 j in that shortest path between the Surfing node 5100 b and thePool node 5100 g (e.g., the product of the 0.9 weight of relationship5102 b, the 0.8 weight of relationship 5102 i and the 0.8 weight ofrelationship 5102 j).

Thus, as will be recalled from above, to score a travel related item(e.g., destination or hotel) with respect to a concept the searchmodules 624, 644 may determine the shortest path from the noderepresenting the item (e.g., destination or hotel) to the noderepresenting that concept and a score associated with that shortestpath. In some embodiments then, to determine such a shortest path andrelated weight it can be determined if there is a metaRelatedrelationship between the concept node and the node representing thetravel related item, and the score associated with such a metaRelatedrelationship. In some embodiments as there may be a maximum “depth” ornumber of hops that a shortest path may have (e.g., 2, 3 or 4 hops) ametaRelated relationship between two nodes will only be establishedbetween two nodes if the shortest path between the two nodes contains anequal or lesser number of hops than this maximum depth.

In this manner, the determination of every path between the concept nodeand the node for the travel related item to determine the shortest pathbetween them may be avoided when a request is received at a dataservice, along with the avoidance of the calculation of the degree ofaffinity of the shortest path. Consequently, the number of processorcycles and amount of memory utilized to make such determinations may beseverely reduced. Moreover, the speed with which the determination oftravel related items and associated content responsive to a requestreceived at a data service are made may be markedly increased.

As the concept networks 612, 614 may be updated or otherwise change at afairly rapid pace, in one embodiment, metaRelated service 670 mayoperate to update the concept networks 612, 614 with these metaRelatedrelationships. The metaRelated service 670 may operate asynchronously atsome time interval (e.g., daily, weekly, etc.) to determine the shortestpath with the highest score between each concept (or other type of) nodeand each other node (if a path exists). The metaRelated service 670 maythen determine if the shortest path is less or equal to a maximum numberof hops. If the shortest path contains a number of hops less than orequal to a maximum depth the metaRelated service may update the conceptnetwork 612, 614 with a metaRelationship between the nodes and assignthe highest score to this metaRelationship. This update may includecreating a new metaRelationship, updating a previously establishedmetaRelationship between the nodes or removing a previously establishedmetaRelationship between the nodes.

As has been discussed, it is desired to provide accurate and responsivetravel related items to a user of embodiments of a chat artificialintelligence systems disclosed herein. Accordingly the accuracy of theconcept networks used in such artificial intelligence systems isimportant, as the accuracy of the travel related items and associatedcontent returned to the user is heavily premised on such conceptnetworks. It is therefore desirable to continually refine the nodes andrelationships (and weights) of the concept networks utilized based onavailable data. As there may be many users interacting with the travelrelated artificial intelligence systems and the searches, activities andinteraction with the travel related chat artificial intelligence systemsand its interfaces may be tracked it may be desirable to utilize suchdata in the refinement of the concepts network used by such artificialintelligence systems.

In one embodiment, then, the destination concept network 612 or hotelconcept network 614 may be automatically updated by concept networkupdater 660. The concept network updater 660 may apply a set of rules,including a machine learning algorithm or the like. These rules can takeinto account queries for particular concepts received through the searchinterfaces 624, 644 of the data services 650, the number of timesparticular paths through the concept network are traversed whensearching the concept networks 612, 614, or user interactions withcontent that is presented to the user (e.g., as ascertained from webtracking data associated with interfaces presented from the artificialintelligence system to the user). These rules can serve to add newconcept nodes to the concept networks 612, 614, establish new edges(relationships) or weights between concept nodes and other nodes, oradjust the weights on existing relationships.

Examples of such rules may be:

-   -   If a user likes “ThumbsUp” content related to a search result        apply a boost (e.g., increase the weight of a relationship)        between all concepts associated with a user interaction on and        the travel item (e.g., hotel, destination, etc.) in the result        that they liked.    -   If a user dislikes “ThumbsDown” content related to a search        result apply a decrement (e.g., decrease the weight of a        relationship) between all concepts associated with a user        interaction on and the travel item (e.g., hotel, destination,        etc.) in the result that they disliked.    -   If a user clicks a search result, apply a boost (e.g., increase        the weight of a relationship) between all concepts associated        with a user interaction searched on and the travel item (e.g.,        hotel, destination, etc.) in the result    -   If a user never clicks on a search result in the result set,        then apply a decrement (e.g., decrease the weight of a        relationship) between all concepts associated with a user        interaction and the search results.

As mentioned above, embodiments may utilize or create an APImicroservice capable of receiving concepts and returning destinations orreceiving a destination and responding with concepts. Generally, asystem extracts concepts from a chat system and uses AI systems (such asthose described above) to recommend one or more destinations based onthe extracted information. For example, assume that an extractionservice extracts several concepts from the chat system. The systems usethe extracted concepts to determine relevant destinations and points ofinterest based on the extracted concepts. Other factors besides theconcepts can also be used, such as geo shapes, weather information,dates, etc. For example, with respect to the concept “skiing”, thesystem may also use weather and travel date information to automaticallyrecommend destinations. For example, if a user wants to go skiing duringmonth A, it would not be desirable to recommend a destination that istoo warm for skiing during month A, even though that destination mayscore high during other months. In another example, if the concept“honeymoon” is extracted, destination recommendations may includedestinations that score high in related concepts such as “romanticgetaway”, “anniversary”, etc., or other concepts mapped to the conceptof “honeymoon”. The destinations that are recommended by the system canbe ranked based on a scoring system, as described herein.

With respect to scoring potential destinations, in some embodiments, thescoring process can be thought of as a matrix of trie concepts versusregions. For the purposes of this description, a trie can be thought ofas a search tree data structure used to store a dynamic set orassociative array of concepts, as described above. Following is Table 1,which illustrates a partial view of a simplified matrix. In thisexample, each row of the matrix represents a region—in this example, thecities City1, City2, and City3. In this example, each column of thematrix represents a trie concept—in this example, hiking, beach, andbars. For the purposes of this discussion, hypothetical scores areentered for each concept and city. Note that the scores can bedetermined by any desired method. For example, scores from third parties(e.g., Triposo—www.triposo.com) can be used and normalized.

In the example of Table 1, assume that a chat system extracts theconcepts “Hiking” (column 1) and “Beach” (column 2) from interactionswith a user. In this example, City3 would score the highest, since City3 has high scores for both concepts. City2 or City1 would score second,depending on the scoring logic used. In one example, the average conceptscores are used. In this example, City2 would rank second to City3 basedon the average concept scores. As discussed above, the scores can becalculated in any desired way. Also note that this is a very simpleexample. In other examples, multiple matrices may be used and morecomplicated scoring logic may be used. Note that different concepts canalso be weighted to get desired results. For example, perhaps “Hiking”is considered more important than “Beach”. In that example, “Hiking” canbe weighted more than “Beach” to get the desired results (i.e., aweighted sum). Therefore, a score for an unimportant concept will notgreatly affect the overall score for a destination.

TABLE 1 Hiking Beach Bars City1 0.7 0.1 0.9 City2 0.3 0.7 0.9 City3 0.90.9 0.5

In one embodiment, to facilitate the recommendation or search fordestinations or hotel, certain information may be determined (e.g.,initially or at certain intervals) such that it may be available for usein the determination of an overall score for a destination or hotel. Forexample, in one embodiment concept net updater 660 may, in oneembodiment, determine concept net relationship matrix 696. This conceptnet relationship matrix may be a matrix M×M matrix where M is equal tothe number of concepts being utilized by the chat system 600. Toillustrate, as discussed above, in one embodiment, there may a singleconcept network that includes all concepts utilized by the chat system600 and separate destination listings and hotel listings may bemaintained (or may be combined into a single listing). In this manner,there may be a set of canonical or complete set of concepts that may beutilized by the chat system 600 for both destinations and hotels.Concept net updater 660 may utilize such a single concept network todetermine a score for the relationship between each concept and everyother concept of the (e.g., complete) set of concepts.

Such a score may, for example be based on a score (e.g., a metaRelatedscore) between the nodes representing the two concepts in the conceptnetwork. The score may also be based on a navigational heuristic. As butone example, scores can be generated between any two concepts in a triebased on the number of edges (or distance) between two concepts, thenumber of levels of the trie that must be traversed, the direction oftraversal (e.g., either up the trie or down the trie) to navigatebetween the two nodes representing those concepts or other traversal ortraversal distance based criteria. In embodiments of such a matrix, thescore between the same concept may be represented by an identity value,such as “1” or any other suitable value. Accordingly, the concept netrelationship matrix 696 may resemble the following matrix:

Concept1 (C1) Concept2 (C2) . . . ConceptN (CN) Concept1 (C1) IdentityScore C1toC2 Score . . . C1toCN Score Concept2 (C2) C2toC1 ScoreIdentity Score . . . C2toCNScore . . . . . . . . . . . . . . . ConceptN(CN) CNtoC1 Score CNtoC2 Score . . . Identity Score

Additionally, a score for each, or a subset of, the concepts, (e.g., theset of canonical or complete set of concepts that may be utilized by thechat system 600) may be determined for each of the destinations storedin the list of destinations 692 or hotels in the listing of hotels 694.In one embodiment, the score for a destination or hotel and the set ofconcepts for that destination or hotel (which may be the same as, orfewer concepts than the complete set of concepts) may be determined bythe respective service (e.g., destination data service 640 or hotel dataservice 620).

To determine such a score for the set of concepts for a destination orhotel, a set of content (documents) may be obtained for the destinationor hotel. Each of these documents may be obtained from a particularsource (e.g., Trip Advisor, Trivago, Wkipedia, etc.) or may be obtainedthrough a search based on the destination or hotel performed through anInternet based search engine such as Google or Bing, or in anothermanner.

For each document, the plain text or other data of the document may beextracted from the main content/body of the document. From this plaintext or other data, the set of concepts that are present in the documentmay be extracted. This extraction may extract and determine a score foreach of the canonical set of concepts utilized by the chat system 600that is present in that document. Scores for those canonical conceptsthat are not present in the document (e.g., cannot be extracted from thedocument) may not be given for that document or may be given a defaultscore such as zero. This extraction and concept score for those conceptspresent in the document may, for example, be determined using termfrequency-inverse document frequency (TFIDF) or the like. In oneembodiment, the document (or an identifier for the document) may bestored by the chat system 600 along with the concept vector for thatdocument (e.g., a vector of concept scores for each (or a subset) of thecanonical set of concepts determined from that document, referred to asa document concept vector).

Accordingly, there may be a set of documents associated with adestination or hotel, where each document has an associated conceptvector denoting a score for each (or a subset of) the canonical set ofconcepts utilized by the chat system 600. The scores for the conceptsfor each of the set of documents can then be combined to create aconcept vector for the set of documents (referred to as an overalldocument concept vector). In other words, in one embodiment, thedocument concept vectors for each of the set of document for adestination can be combined to generate a score for each (or a subsetof) the canonical set of concepts utilized by the chat system 600. Thescore for a concept in the overall document vector may be determined forexample by determining the score for that concept in each documentvector where it is present and applying a heuristic or weight to thescores as determined from each document concept vector. This weight may,for example, be a dual weighted average of the source of the respectivedocument based on a trust level for that source. For any concepts in thecanonical set of concepts for which no score was present in an of thedocument concept vectors for documents determined for that destinationor hotel a default value (e.g., zero or a relatively low value) may beutilized.

Once the overall document concept vector is determined, a destination(or hotel) concept vector may be created, where the destination conceptvector may include a score for that destination for each (or a subsetof) the canonical set of concepts used by the chat system 600. In oneembodiment, a destination (or hotel) concept vector for a destination(or hotel) can be determined by performing a matrix multiplicationoperation between the concept net relationship matrix 696 created forthe canonical set of concepts of the chat system 600 and the overalldocument concept vector determined for the destination (or hotel). Sucha matrix multiplication produces a destination (or hotel) concept vectorhaving a score for each (or a subset of) the canonical set of conceptsthat is reflective of both the concepts and scores extracted from thedocuments obtained for the destination (or hotel) and the relationshipsthat each of the canonical set of concepts of the chat system 600 haveto one another.

The destination (or hotel) concept vector determined for a destination(or hotel) can be stored in association with that destination (or hotel)in the list of destination 692 (or list of hotels 694). In oneembodiment, for example, the destination (or hotel) concept vector maybe stored as key (concept) value (score) pairs. This destination (orhotel) concept vector can then be used by the search module 644 (orsearch module 624) when performing a search for a destination (or hotel)related to one or more concepts. More specifically, when a request(search) for a set of destinations (or hotels) associated with one ormore concepts is received by a search module 624, 644 a search ofdestinations or hotels can be based on the score for the received one ormore concepts in the destination (or hotel) concept vectors for the listof destinations (or hotels).

For example, a search could be conducted for a set of destinations inthe list of destinations 692 based on maximizing the combined scores foreach of the set of received concepts as determined from the destinationconcept vectors associated with each of the set of destinations. Inparticular, for each destination in the list of destinations 692 theassociated destination concept vector may be obtained and the score foreach of the received concepts for the search determined for thatdestination from that destination's destination concept vector. Thesescores (e.g., the scores for each concept of the received concept asstored in the destination concept vector for that destination) can becombined (e.g., added, averaged, the maximum score selected orotherwise) to determine a score for that destination for the search. Anumber of destinations with the highest score for that search can thenbe returned.

Following are more detailed examples of some embodiments of methods thatmay be employed by AI system capable of returning recommendations ofdestinations (or hotels) based on concepts extracted from embodiments ofthe chat system described above or from another source. The selection ofimages corresponding to these destinations (or hotels) is alsodescribed. While embodiments may be described with respect todestinations, it will be understood that such embodiments may also applyto hotels or other real-estate or location related contexts such asapartments, houses, points of interest, tours or the like.

Looking then at FIG. 8, a flow diagram for one method of searching fordestinations is depicted. Initially, a concept net relationship matrixmay be determined (STEP 750). This concept net relationship may bedetermined (e.g., initially or at certain intervals) such that it may beavailable for use in the determination of an overall score for adestination or hotel. This concept net relationship matrix may be amatrix M×M matrix where M is equal to the number of concepts beingutilized by a system. To illustrate, as discussed above, in oneembodiment, there may a single concept network that includes allconcepts utilized and separate destination listings may be maintained.In this manner, there may be a set of canonical or complete set ofconcepts that may be utilized for destinations. This concept network maybe utilized to determine a score for the relationship between eachconcept and every other concept of the set of concepts.

Such a score may, for example be based on a score (e.g., a metaRelatedscore) between the nodes representing the two concepts in the conceptnetwork. The score may also be based on a navigational heuristic. As butone example, scores can be generated between any two concepts in a triebased on the number of edges (or distance) between two concepts, thenumber of levels of the trie that must be traversed, the direction oftraversal (e.g., either up the trie or down the trie) to navigatebetween the two nodes representing those concepts or other traversal ortraversal distance based criteria. In embodiments of such a concept netrelationship matrix, the score between the same concept may berepresented by an identity value, such as “1” or any other suitablevalue.

Additionally, a score for each, or a subset of, the concepts, (e.g., theset of canonical or complete set of concepts) may be determined for eachof the destinations stored in the list of destinations (LOOP 752). Thus,for a destination a set of content (documents) may be determined forthat destination (STEP 754). Each of these documents may be obtainedfrom a particular source (e.g., Trip Advisor, Trivago, Wikipedia, etc.)or may be obtained through a search based on the destination or hotelperformed through an Internet based search engine such as Google orBing, or in another manner.

For each document, a document concept vector can be determined (STEP756). In one embodiment, for the document, the plain text or other dataof the document may be extracted from the main content/body of thedocument. From this plain text or other data, the set of concepts thatare present in the document may be extracted. This extraction mayextract and determine a score for each of the canonical set of conceptsutilized by the system that is present in that document. Scores forthose canonical concepts that are not present in the document (e.g.,cannot be extracted from the document) may not be given for thatdocument or may be given a default score such as zero. This extractionand concept score for those concepts present in the document may, forexample, be determined using term frequency-inverse document frequency(TFIDF) or the like. The scores for each of the concepts for thatdocument may be stored or other used as the document concept vector forthat document.

Once it is determined that the document concept vectors have beendetermined for each of the documents for the destination (Y Branch ofSTEP 758) an overall document concept vector may be determined for thedestination (STEP 760). Specifically, there may be a set of documentsassociated with the destination, where each document has an associatedconcept vector denoting a score for each (or a subset of) the canonicalset of concepts utilized by the system. The scores for the concepts foreach of the set of documents can be combined to create the overalldocument concept vector). In one embodiment, the document conceptvectors for each of the documents for a destination can be combined togenerate a score for each (or a subset of) the canonical set ofconcepts. The score for a concept in the overall document vector for adestination may be determined, for example, by determining the score forthat concept in each document vector where it is present and applying aheuristic or weight to the scores as determined from each documentconcept vector. This weight may, for example, be a dual weighted averageof the source of the respective document based on a trust level for thatsource. For any concepts in the canonical set of concepts for which noscore was present in an of the document concept vectors for documentsdetermined for that destination or hotel a default value (e.g., zero ora relatively low value) may be utilized.

Once the overall document concept vector is determined, a destinationconcept vector may be created, where the destination concept vector mayinclude a score for that destination for each (or a subset of) thecanonical set of concepts (STEP 762). In one embodiment, a destinationconcept vector for a destination can be determined by performing amatrix multiplication operation between the concept net relationshipmatrix created for the canonical set of concepts and the overalldocument concept vector determined for the destination. Such a matrixmultiplication produces a destination concept vector having a score foreach (or a subset of) the canonical set of concepts that is reflectiveof both the concepts and scores extracted from the documents obtainedfor the destination and the relationships that each of the canonical setof concepts of the system have to one another.

The destination concept vector determined for a destination can bestored in association with that destination. In one embodiment, forexample, the destination concept vector may be stored as key (concept)value (score) pairs. This destination concept vector can then be usedwhen performing a search for a destination (or hotel) related to one ormore concepts. More specifically, when a request (search) for a set ofdestinations associated with one or more concepts is received (STEP764), the set of concepts for the search may be determined from therequest (STEP 766). A search of destinations can be based on the scorefor the received one or more concepts in the destination concept vectorsfor the destinations.

Specifically, in one embodiment, each destination may be scored based onthe set of concepts for the search and the destination concept vectorfor that destination (STEP 768). For example, a search could beconducted for a set of destinations in the list of destinations based onmaximizing the combined scores for each of the set of received conceptsas determined from the destination concept vectors associated with eachof the set of destinations. In particular, for each destination in thelist of destinations the associated destination concept vector may beobtained and the score for each of the received concepts for the searchdetermined for that destination from that destination's destinationconcept vector. These scores (e.g., the scores for each concept of thereceived concept as stored in the destination concept vector for thatdestination) can be combined (e.g., added, averaged, the maximum scoreselected or otherwise) to determine a score for that destination for thesearch. A number of destinations with the highest score for that searchcan then be returned (STEP 770).

FIG. 9 is a flow diagram illustrating another embodiment of a method forrecommending destinations. A search query is created for each concept(STEP 710). For example, the search query may be “top world destinationsfor {concept}”, for each respective trie concept. In one example, thiscan be accomplished using the Bing Web Search API to get the top 50search results, in the form of 50 URLs (STEP 712). Details of the BingWeb Search API can be found at[https://azure.microsoft.com/en-us/services/cognitive-services/bing-web-search-api/],which is incorporated by reference herein. The Bing Web Search APIenables the retrieval of web documents indexed by Bing with resultsnarrowed down by result type, freshness, and more. The search resultsare filtered (e.g., using whitelisted URLs/concepts) to validate theresults (STEP 714). Various validation/filtering techniques may be used.For example, some URLs may be considered valid and whitelisted (e.g.,www.travel.com). In another example, if the text of a respective conceptis present in the URL, that may be an indication that this result isvalid. For example, with respect to the concept “skiing”, if a URL hasthe work “ski” or “skiing” in it, that may be an indication that thesource is valid.

For each filtered website, plain text of the website is extracted fromthe main content/body of the website using a web scraper (STEP 716). Anydesired web scraping tool(s) may be used. Exemplary web scraping toolsinclude BeautifulSoup, Readability, etc. At this point, the plain textfor each of the filtered web sites is available to the system. From thisplain text, all “location” entities are extracted from the page text ofeach respective web page (STEP 718). In one example, this is done usingthe Google Cloud Natural Language API, which can return a list of uniquelocation identifiers for any geographic entity referenced in the text,such as cities, states, countries, etc. In some embodiments, for eachwebsite, the system collects all of the entities found, and keeps trackof the frequency count, to help determine a score for the respectiveentity (STEP 720).

From all of this data, frequencies can be normalized and utilized toinitialize a matrix (STEP 722). In some embodiments, this matrix mayinclude one row for each region and one column for each concept, similarto Table 1 above. Note that in some embodiments, various matrices can bederived from data sources, such as the Internet (i.e., a scrapermatrix), third party data providers such as Triposo (i.e., a Triposomatrix) or Wkivoyage (i.e., a Wikivoyage matrix), etc. Such matrices mayeach be, like those described above, a concept by region matrix using arelated methodology based on frequency counts. In some embodiments, thescraper matrix is merged with the Triposo matrix (STEP 724) (or anyother matrices, such as the Wkivoyage matrix, etc.). Note that thematrices are not merged at query time, but beforehand, offline. The endresult is a unified matrix with scores for each concept/destinationbased on all of the various matrices. Note also that different matricesfor different contexts may have different weights, based on specifics ofa particular context.

At this point, the system has a unified matrix, and the system needs tobe able to search concepts. Next, tables are created in a database(e.g., DynamoDB). For example, refer to the database tables shown inFIGS. 11-12. A separate process moves the data from the database (e.g.,DynamoDB) tables into Elasticsearch. The data is then merged with datafrom other sources stored in a number of different databases (e.g.,weather data from a Postgres database) (STEP 726). (See, e.g., FIGS.10-12, described below, for exemplary DynamoDB and Elasticsearchscreenshots). Next, this other data and the data from the unified tableare put together in Elasticsearch to create a searchable database thatcan be indexed by concept, region, name, etc. Finally, as shown in FIG.9, the Destination AI API is uses Elasticsearch query logic todynamically deliver results (STEP 728).

Note that in Elasticsearch, searches can be filtered based on weather,geometry, coordinates, etc. For example, if a user only wants to go tothe United States, the system can filter out results outside of theUnited States. Similarly, if a user wants to go somewhere warm inJanuary, regions that are not warm in January can be filtered out.Results can also be filtered using client filters. For example, if theweb publisher (client) is an airline, destinations that are not inregions where they fly can be filtered out.

Following is a brief overview of how the Destination AI querying logicworks in some embodiments. In one embodiment, a search may be based onthe unified concept score matrix described above. As described above, insome examples, the scores in the unified matrix are derived from Tripososcores (or other third party sources) and the web scraper. In someexamples, for a single incoming concept, the system simply finds thecity that has the highest score for the input concept. For multipleconcepts, an example input query could look like the following URL(specifying concepts “hiking” and “wine”):

https://dest-api.test.wayblazer.systems/destinations?x-api-key=e8824b30-1a48-11e7-bf11-81cbd96aa5d3&concept=Hiking&concept=Wine

In some embodiments, at search time, the system finds the destinationthat has the highest weighted sum of all concept scores and theircorresponding concept weights. Following is the relevant part of anexemplary search query for the example above:

“sort”: { “_script”: { “order”: “desc”, “script”: { “inline”:“(doc[‘concepts.Hiking’].value + doc[‘concepts.Wine’].value) / 2”“lang”: “painless” }, “type”: “number” } }

This exemplary approach is simple and works fairly well, but there arescenarios where more sophisticated logic may be desired. For example, ifone destination has a really high score for hiking but not for wine, itmight still outrank a different destination with medium scores for both.

For example, look at two hypothetical destinations (destination_1 anddestination_2) and their respective scores:

destination_1: Hiking is 10, Wine is 0, average is 5

destination_2: Hiking is 4, Wne is 4, average is 4

In this case, looking only at the average scores of the two concepts,the system would recommend destination_1 over destination_2, even thoughdestination_2 might actually be the better result. Therefore, the systemcan be configured to recommend destinations using more sophisticatedlogic, as desired.

FIGS. 10A-12B are search query and database screenshots for exemplaryembodiments of the Destination AI API discussed above. FIGS. 10A, 10Band 10C are an exemplary screenshot of an Elasticsearch query for theregion of Cape Town. As shown, various parameters are listed, includingconcepts and their associated scores. FIGS. 11A and 11B are an exemplaryscreenshot showing an overview of a DynamoDB database table. As shown,the database includes various columns, including wiki_id, concepts,country_name, country_wiki_id, etc. FIGS. 12a and 12B are an exemplaryscreenshot showing details of the DynamoDB database, showing variouscolumns in the database and a column show/hide dialog box.

As mentioned above, once one or more destinations or hotels aredetermined, one or more images corresponding to each destination (orhotel) may be selected for display to users (for example in the carddisplay areas 130, 132 shown in FIGS. 1A-1D). The images selected for agiven destination (or hotel) can be selected in any desired manner. Inone embodiment, the images selected may be based on concepts (e.g., asreceived in a request to a data service). However, even when no conceptsare utilized, desired images may be selected for display. When selectingimages, various considerations may be taken into account. For example,it may be desirable to select a variety of images, so that a group ofvery similar images are not selected. While images can be selected usingany many types of concepts, it may be desired to also include“important” concepts, such as architecture, museums, etc., so thatselected images that relate to desired concepts also relate majorconcepts that would be generally associated with the destination. It mayalso be desirable to be able to select images even when an acquiredimage is not tagged with a specific concept, so long as the image can bemapped or associated with a related concept (described below). It mayalso be desirable to include within a group of selected images, imagesdepicting certain “default” or “gold standard” items relating toparticular concepts for a particular destination. As would be evident toone skilled in the art, the selection of images corresponding to adestination can take into account numerous considerations, as desired.

FIG. 13 is a flow diagram depicting one embodiment of a method 1300 forselecting and ranking images. FIG. 14 is a flow diagram depicting oneembodiment of a method 1400 for ranking images where no concepts areidentified (e.g., by the chat system). Numerous other examples are alsopossible, as one skilled in the art would understand. First, one or moredestinations are determined, for example, based on information extractedfrom a chat system (described in detail above) (STEP 1310). For eachdestination, one or more images are acquired (STEP 1320). Images can beacquired in any desired manner. For example, images can be acquired froman existing database, obtained from third parties, etc. In one example,10 images are acquired from each of three third party sources, such aspixabay (https://pixabay.com/), Flickr (https://www.flickr.com/), andTriposo (https://www.triposo.com/). These particular examples provide anapplication programming interface (API) to enable automated access totheir stocks of images. In this way, images can be retrievedautomatically by sending a query to the site. The query may include thedestination or the destination, along with one or more conceptsidentified by the chat system.

Once a set of images are acquired, each of the images are analyzed todetermine what is depicted in each respective image. The image analysiscan be accomplished in any desired manner. In one example, a third partyservice, such as the service provided by Clarifai(https://clarifai.com/) is used to analyze images. In particular,concepts may be identified in each of the images (STEP 1330). For eachconcept identified, a score (e.g., probability) is calculated (STEP1340). Such a score may relate to the probability that a concept isdepicted in, or related to, the image. It will be noted that the imageacquisition and determination of concepts or scores may be determinedbefore a request is received by a data service and may be stored in adata store utilized by the system (e.g., chat system).

FIG. 15 illustrates an example of an image that has been analyzed by aservice such as Clarifai to identify concepts and probabilities. In theexample shown, an image 1500 of Austin, Tex. is analyzed. In response, aplurality of predicted concepts 1510 are listed, and for each concept1510, a probability 1520 is shown. In the example shown, the concept“city” is predicted with a probability of 0.988, meaning that it ishighly likely the image 1500 depicts a city. Other concepts, forexample, “river” (probability of 0.885), is most likely depicted in theimage, but not with as high of a certainty as “city.” In accordance withthis example, then, such an image may be stored in association with thedestination “Austin, Tex.”, associated with the concepts “city” and thescore 0.988 and the concept “river” and the associated score 0.855.

Returning to FIG. 13, once the predicted concepts and correspondingscores are determined for an image, the predicted concepts andcorresponding scores are converted or mapped (if necessary) to conformto the concepts system used to select and rank the images (STEP 1350).These may be, for example, the set of concepts included in a concept netrelationship matrix for the system.

In particular, it is possible that the predicted concepts from a thirdparty service (such as Clarifai) will not exactly match the set ofconcepts used by the system or used to rank images. For example,concepts used in the concept net relationship matrix or the matrix oftrie scores as discussed above may not all match concepts from Clarifai.In some examples, an unmatched Clarifai concept may have a synonym trieconcept (e.g., may be stored as a synonym of a concept of the trie). Inthat case, the concept can be converted (renamed) to match the trieconcept. In some examples, an unmatched Clarifai concept can be ignoredif there is no close synonym, or if it is a concept that is not desired.With respect to the probabilities provided by Clarifai, a concept scorein the trie or the concept net relationship matrix can correspond to theClarifai probability, or can be modified, as desired.

In one embodiment, once a set of concept and associated scores aredetermined for an image, an image concept vector may be determined forthe image using the concept net vector matrix. Such an image conceptvector may include a score for each of the concepts as utilized by thesystem (e.g., as included in the concept net vector matrix). This may bedetermined by performing vector multiplication between a column or rowin the concept net vector matrix corresponding to a concept for which ascore has been determined (e.g. “city” or “river” in the example above).In this manner, the consanguinity between a determined concept for theimage and the other concepts as utilized by the system may bedetermined. In another embodiment, some combination of the scores forthe concepts for which a score has been determined for the image (e.g.“city” or “river” in the example above) may be determined. This combinedscore can then be used in association with the scores between thosedetermined concepts and every other concept in the concept net vectormatrix. For example, to determine a score for the image for a particularundetermined concept, the score between each determined concept and theundetermined concept can be determined from the concept net vectormatrix. The combined score determined from the scores for the determinedconcept can then be combined (e.g., multiplied) with the scoresdetermined between each determined concept and the undetermined conceptto determine a score for the undetermined concept for that image. Thescores determined for each determined concept for the image can becombined with the scores determined for each undetermined concept forthe image to form the image concept vector.

Accordingly, after the images are analyzed, the set of images for agiven destination are ranked and selected (STEP 1360). The ranking ofimages can be based on any criteria, depending information available anddesired results. In one example, images are ranked based on conceptsreceived in the request. Generally, if the system determines that a useris interested in a particular concept (e.g., “surfing”), images can beranked based on the “surfing” concept score associated with each image(e.g., as determined from the image concept vector associated with theimage). Where multiple concepts are identified by the system (e.g.,based on a user interaction or as submitted to a destination dataservice), the concept scores can be added for ranking purposes. Ifdesired, concept scores are weighted, so that important concepts have ahigher effect on ranking than less important concepts. For example,certain concepts can be considered to be important, such as“architecture”, “museums”, etc., versus less important concepts such as“fine dining”, “animals”, etc. If a user is interested in “fine dining”,it is more desirable to provide an image of food with the city skylineor a gorgeous restaurant standing in the center of the city, versus aclose-up view of food. Therefore, while the concept “fine dining” isstill considered in the ranking, it is considered along with theimportant concepts. This can be accomplished using weights to rankimages having scores for both the identified concepts and the importantconcepts.

In another embodiment, where no concepts are provided by the chatsystem, the images can be ranked as depicted in the flow diagram of FIG.14. Note that the embodiment of the method for image selection depictedin FIG. 14 could also be used when concepts are provided. In thisembodiment, images may be ranked based on a weighted sum of the trie orconcept scores, where “important” concepts are given more weight thanless important concepts. However, as mentioned above, it may not bedesired to end up with a group of very similar images. For example, ifthe destination is “San Francisco”, it may not be desired to display 30images of the Golden Gate Bridge. Instead, it may be desired to includea variety of images, even if it means images with higher trie scores areranked lower than other images. The embodiment as depicted in FIG. 14may be utilized to obtain such a variety of images.

Initially, the images are ranked based on scores (STEP 1410), asdescribed above. Based on the scores, a first image is selected (STEP1420). Once a first image is selected, the system determines whetherother images are similar to the first selected image. This determinationcan be accomplished in any desired manner. Similarity can be determinedby comparing trie scores or concept net vector matrix scores of theimportant concepts. The more similar an image is to the first selectedimage (e.g., when an image has similar scores of the importantconcepts), the more it will be penalized for the purpose of selectingthe second image. For example, assume image1 was selected based on highscores for “architecture” and “museums”. If image2 has the same orsimilar scores for “architecture” and “museums”, it will be penalizedmore than image3 that scores low for “architecture” and “museums”. Inone example, images are penalized by subtracting a number from itsweighted sum of trie scores. Accordingly, a penalty is selectivelyapplied to the remaining (the non-selected) images based on theirsimilarity to the previously selected image(s) (STEP 1430). In oneexample, the penalty can be a constant number applied to images thatmeet a similarity threshold. In another example, the penalty can varydepending on the degree of similarity.

Once the penalty is applied to certain remaining images, all of theremaining images are re-ranked, taking the penalty into account (STEP1440). Based on the re-ranking, a second image is selected (STEP 1450).It can then be determined if enough images have been selected (STEP1460). If so, the method ends. If not, the method repeats(STEPS1430-1460) until a desired number of images are selected. Forexample, the second time through the loop, a similarity is determinedbetween the remaining images and the previously selected image (in thiscase, the first image) (STEP 1430). Similarity can be determined in anydesired manner. In one example, the remaining images (with theiradjusted scores from the first penalty application) are compared forsimilarity to the second selected image and penalized based on theirsimilarity to the second image. In another example, the scores of someor all previously selected images can be averaged, with similarity basedon the average scores. In any event, each time through the loop (STEPS1430-1460), the scores may be re-adjusted and the images re-ranked. Theend result is that the selected images will be diverse, while stillbeing based on high scores (e.g., for concepts being utilized to scorethe images, such as those received in a request for a destination orimage).

In some examples, a user may be searching for a concept that is notassociated with one or more of the images for the relevant destination.For example, assume a search is being conducted for the concept “lake”,but none of the images are tagged with “lake.” It may be desirable toinclude an image of a beach, since “lake” and “beach” are similar. Thiscan be accomplished using the concepts as found in the concept network.As discussed above, a concept network, such as a trie, can be thought ofas a search tree structure used to store a dynamic set or associativearray of concepts. For example, a trie will have various parent nodesand child nodes. In the example above, the trie may have a “water” nodewith “lake” and “beach” being child nodes to “water.”

FIG. 16 is a diagram depicting a portion of a trie with nodes forconcepts “water,” “lake,” and “beach.” As shown, water node 1610 is aparent to lake node 1620 and beach node 1630. FIG. 16 also shows linesillustrating the relation (e.g., bi-directional) between the parent node1610 and the child nodes 1620 and 1630. Such relations may, or may not,be included in a trie, and may be determined based on a variety ofcriteria or heuristics as discussed. Generally, a child node will bevery related to a parent node (thus the “distance” factor 0.1). Forexample, a picture of a lake or a beach is highly likely to includewater. The “distance” from the parent node 1610 to the child nodes 1620and 1630 are longer, since they are less related. For example, a pictureof water does not necessarily include a lake or beach, but could. Inother words, if the concept is “lake”, you may not necessarily want animage of “water”, but if the concept is “water,” then it may beacceptable to return an image of a lake. The distances 0.5 and 0.1 aremerely exemplary, for the purposes of discussion.

In the example discussed above, it was assumed that a search is beingconducted for the concept “lake”, but none of the images are tagged with“lake.” Although none of the images are tagged with the concept “lake,”the system can still generate a score for “lake.” Concepts that areincluded in an image can be mapped to “lake”, and the score of theshortest path can be used. Note that FIG. 16 is only a partial view ofan example trie used for illustrative purposes, and that an actual triemay include all concepts utilized by a system or multiple tries may beused, etc. In this example, the shortest path to lake node 1620 is frombeach node 1630, through parent water node 1610, to lake node 1620. A“lake” score can be calculated by adding the total distance from beachnode 1630 to lake node 1620. In this example, the distance from beachnode 1630 to lake node 1620 is 0.5 plus 0.1, totaling a distance of 0.6.So, if a user searches for a lake image, and the system cannot find alake image, it will try to find a water image. If the system cannot finda water image, it may provide a beach image.

In the example above, assume that the beach image has a beach conceptscore of 0.9. The system will convert that score to a lake concept scoreusing the calculated distance. In one example, the system a matrix iscreated for each image that will have scores for the concepts, even whenClarifai did not include a concept and probability/score. For example,the concept net relationship matrix may be used. As another example, amatrix is populated as discussed above, using the shortest distancebetween a missing score and an existing score (like with concepts lakeand beach above).

In one example, a trie includes on the order of 1400 concepts, so thematrix will be 1400×1400 in size, with a distance value for eachpossible pair of concepts as discussed. When the scores are put together(for example, adding the lake score in the above example) the end resultis like a weighted sum. In our example, for concept beach, the distanceis 0.0, since beach has a score. In this case, for beach, the otherdistances are meaningless and ignored. In our example, the lake scorewill be equal to the beach score, multiplied by the distance (in thiscase 0.6).

In some examples, each time an image is ingested, the matrix isgenerated, so you have a fully populated matrix with scores for everyconcept (for example, including the lake concept discussed above). Insome examples, these matrices are stored in Elasticsearch.

Note that the matrix becomes a combination of variables for the x and yconcepts. By tweaking the x and y values of concepts, you can tweak thefinal scores to make the results as close to desired as possible. By “asclose as desired”, it is meant that sometimes, an implementation willneed a “gold standard.” For example, when a system is implemented,certain “gold standards” may be desired. For example, for destination“City1”, it may be desired that the results should include an image of“Museum1”. The various weights and scores can be tweaked such that animage that includes “Museum1” likely to be included in the results.

Reference is now made to the Appendices which may aid in understandingof certain embodiments. Appendices B and C show examples of APIqueries/responses for the Destination AI API discussed above. AppendicesB and C show API query/response examples. Each example shows a URL andfull query. Each example also shows the API response (e.g., in a jsonfile).

Appendix B (B1/B2) shows an example of a standard concept search (here,for the concept “Hiking”). In this example, the query (Appendix B1) alsospecifies the region “United States”. As shown, the query includes ageo_shape filter that will limit the API response (Appendix B2) todestinations within the United States.

Appendix C (C1/C2) show an example of details of a query for a knowndestination (here, for the destination “Venice”). As shown, the APIresponse (Appendix C2) includes various concepts and their associatedscores with respect to Venice returned by a destination service.

These, and other, aspects of the disclosure and various features andadvantageous details thereof are explained more fully with reference tothe exemplary, and therefore non-limiting, embodiments illustrated anddetailed in the accompanying appendices. It should be understood,however, that the detailed description and the specific examples, whileindicating the some embodiments, are given by way of illustration onlyand not by way of limitation. Descriptions of known programmingtechniques, computer software, hardware, operating platforms andprotocols may be omitted so as not to unnecessarily obscure thedisclosure in detail. Various substitutions, modifications, additions orrearrangements within the spirit or scope of the underlying inventivesubject matter will become apparent to those skilled in the art fromthis disclosure, including the accompanying appendices.

Although the invention has been described with respect to specificembodiments thereof, these embodiments are merely illustrative, and notrestrictive of the invention. The description herein of illustratedembodiments of the invention, including the description in the Abstractand Summary, is not intended to be exhaustive or to limit the inventionto the precise forms disclosed herein (and in particular, the inclusionof any particular embodiment, feature or function within the Abstract orSummary is not intended to limit the scope of the invention to suchembodiment, feature or function). Rather, the description is intended todescribe illustrative embodiments, features and functions in order toprovide a person of ordinary skill in the art context to understand theinvention without limiting the invention to any particularly describedembodiment, feature or function, including any such embodiment featureor function described in the Abstract or Summary. While specificembodiments of, and examples for, the invention are described herein forillustrative purposes only, various equivalent modifications arepossible within the spirit and scope of the invention, as those skilledin the relevant art will recognize and appreciate. As indicated, thesemodifications may be made to the invention in light of the foregoingdescription of illustrated embodiments of the invention and are to beincluded within the spirit and scope of the invention. Thus, while theinvention has been described herein with reference to particularembodiments thereof, a latitude of modification, various changes andsubstitutions are intended in the foregoing disclosures, and it will beappreciated that in some instances some features of embodiments of theinvention will be employed without a corresponding use of other featureswithout departing from the scope and spirit of the invention as setforth. Therefore, many modifications may be made to adapt a particularsituation or material to the essential scope and spirit of theinvention.

Reference throughout this specification to “one embodiment”, “anembodiment”, or “a specific embodiment” or similar terminology meansthat a particular feature, structure, or characteristic described inconnection with the embodiment is included in at least one embodimentand may not necessarily be present in all embodiments. Thus, respectiveappearances of the phrases “in one embodiment”, “in an embodiment”, or“in a specific embodiment” or similar terminology in various placesthroughout this specification are not necessarily referring to the sameembodiment. Furthermore, the particular features, structures, orcharacteristics of any particular embodiment may be combined in anysuitable manner with one or more other embodiments. It is to beunderstood that other variations and modifications of the embodimentsdescribed and illustrated herein are possible in light of the teachingsherein and are to be considered as part of the spirit and scope of theinvention.

In the description herein, numerous specific details are provided, suchas examples of components or methods, to provide a thoroughunderstanding of embodiments of the invention. One skilled in therelevant art will recognize, however, that an embodiment may be able tobe practiced without one or more of the specific details, or with otherapparatus, systems, assemblies, methods, components, materials, parts,or the like. In other instances, well-known structures, components,systems, materials, or operations are not specifically shown ordescribed in detail to avoid obscuring aspects of embodiments of theinvention. While the invention may be illustrated by using a particularembodiment, this is not and does not limit the invention to anyparticular embodiment and a person of ordinary skill in the art willrecognize that additional embodiments are readily understandable and area part of this invention.

Embodiments discussed herein can be implemented in a computercommunicatively coupled to a network (for example, the Internet),another computer, or in a standalone computer. As is known to thoseskilled in the art, a suitable computer can include a central processingunit (“CPU”), at least one read-only memory (“ROM”), at least one randomaccess memory (“RAM”), at least one hard drive (“HD”), and one or moreinput/output (“I/O”) device(s). The I/O devices can include a keyboard,monitor, printer, electronic pointing device (for example, mouse,trackball, stylus, touch pad, etc.), or the like.

ROM, RAM, and HD are computer memories for storing computer-executableinstructions executable by the CPU or capable of being compiled orinterpreted to be executable by the CPU. Suitable computer-executableinstructions may reside on a computer readable medium (e.g., ROM, RAM,or HD), hardware circuitry or the like, or any combination thereof.Within this disclosure, the term “computer readable medium” is notlimited to ROM, RAM, and HD and can include any type of data storagemedium that can be read by a processor. For example, a computer-readablemedium may refer to a data cartridge, a data backup magnetic tape, afloppy diskette, a flash memory drive, an optical data storage drive, aCD-ROM, ROM, RAM, HD, or the like. The processes described herein may beimplemented in suitable computer-executable instructions that may resideon a computer readable medium (for example, a disk, CD-ROM, a memory,etc.). Alternatively, the computer-executable instructions may be storedas software code components on a direct access storage device array,magnetic tape, floppy diskette, optical storage device, or otherappropriate computer-readable medium or storage device.

Any suitable programming language can be used to implement the routines,methods or programs of embodiments of the invention described herein,including C, C++, Java, JavaScript, HTML, or any other programming orscripting code, etc. Other software/hardware/network architectures maybe used. For example, the functions of the disclosed embodiments may beimplemented on one computer or shared/distributed among two or morecomputers in or across a network. Communications between computersimplementing embodiments can be accomplished using any electronic,optical, radio frequency signals, or other suitable methods and tools ofcommunication in compliance with known network protocols.

Different programming techniques can be employed such as procedural orobject oriented. Any particular routine can execute on a single computerprocessing device or multiple computer processing devices, a singlecomputer processor or multiple computer processors. Data may be storedin a single storage medium or distributed through multiple storagemediums, and may reside in a single database or multiple databases (orother data storage techniques). Although the steps, operations, orcomputations may be presented in a specific order, this order may bechanged in different embodiments. In some embodiments, to the extentmultiple steps are shown as sequential in this specification, somecombination of such steps in alternative embodiments may be performed atthe same time. The sequence of operations described herein can beinterrupted, suspended, or otherwise controlled by another process, suchas an operating system, kernel, etc. The routines can operate in anoperating system environment or as stand-alone routines. Functions,routines, methods, steps and operations described herein can beperformed in hardware, software, firmware or any combination thereof.

Embodiments described herein can be implemented in the form of controllogic in software or hardware or a combination of both. The controllogic may be stored in an information storage medium, such as acomputer-readable medium, as a plurality of instructions adapted todirect an information processing device to perform a set of stepsdisclosed in the various embodiments. Based on the disclosure andteachings provided herein, a person of ordinary skill in the art willappreciate other ways or methods to implement the invention.

It is also within the spirit and scope of the invention to implement insoftware programming or code an of the steps, operations, methods,routines or portions thereof described herein, where such softwareprogramming or code can be stored in a computer-readable medium and canbe operated on by a processor to permit a computer to perform any of thesteps, operations, methods, routines or portions thereof describedherein. The invention may be implemented by using software programmingor code in one or more general purpose digital computers, by usingapplication specific integrated circuits, programmable logic devices,field programmable gate arrays, optical, chemical, biological, quantumor nanoengineered systems, components and mechanisms may be used. Ingeneral, the functions of the invention can be achieved by any means asis known in the art. For example, distributed, or networked systems,components and circuits can be used. In another example, communicationor transfer (or otherwise moving from one place to another) of data maybe wired, wireless, or by any other means.

A “computer-readable medium” may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, system ordevice. The computer readable medium can be, by way of example only butnot by limitation, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, system, device,propagation medium, or computer memory. Such computer-readable mediumshall generally be machine readable and include software programming orcode that can be human readable (e.g., source code) or machine readable(e.g., object code). Examples of non-transitory computer-readable mediacan include random access memories, read-only memories, hard drives,data cartridges, magnetic tapes, floppy diskettes, flash memory drives,optical data storage devices, compact-disc read-only memories, and otherappropriate computer memories and data storage devices. In anillustrative embodiment, some or all of the software components mayreside on a single server computer or on any combination of separateserver computers. As one skilled in the art can appreciate, a computerprogram product implementing an embodiment disclosed herein may compriseone or more non-transitory computer readable media storing computerinstructions translatable by one or more processors in a computingenvironment.

A “processor” includes any, hardware system, mechanism or component thatprocesses data, signals or other information. A processor can include asystem with a general-purpose central processing unit, multipleprocessing units, dedicated circuitry for achieving functionality, orother systems. Processing need not be limited to a geographic location,or have temporal limitations. For example, a processor can perform itsfunctions in “real-time,” “offline,” in a “batch mode,” etc. Portions ofprocessing can be performed at different times and at differentlocations, by different (or the same) processing systems.

It will also be appreciated that one or more of the elements depicted inthe drawings/figures can also be implemented in a more separated orintegrated manner, or even removed or rendered as inoperable in certaincases, as is useful in accordance with a particular application.Additionally, any signal arrows in the drawings/figures should beconsidered only as exemplary, and not limiting, unless otherwisespecifically noted.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having,” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,product, article, or apparatus that comprises a list of elements is notnecessarily limited only those elements but may include other elementsnot expressly listed or inherent to such process, product, article, orapparatus.

Furthermore, the term “or” as used herein is generally intended to mean“and/or” unless otherwise indicated. For example, a condition A or B issatisfied by any one of the following: A is true (or present) and B isfalse (or not present), A is false (or not present) and B is true (orpresent), and both A and B are true (or present). As used herein, a termpreceded by “a” or “an” (and “the” when antecedent basis is “a” or “an”)includes both singular and plural of such term (i.e., that the reference“a” or “an” clearly indicates only the singular or only the plural).Also, as used in the description herein, the meaning of “in” includes“in” and “on” unless the context clearly dictates otherwise.

What is claimed is:
 1. A artificial intelligence system, comprising: aprocessor; and a data store, storing a destination concept vector foreach of a set of destinations and destination content for each of theset of destinations, wherein the destination content for eachdestination includes a set of images for the destination and each of theset of images is associated with an image concept vector; anon-transitory computer-readable medium comprising instructions for:receiving a request from a widget at a user device, the requestincluding a current interaction entered by a user in an interfacedisplayed at the user device; retrieving a session associated with theuser, wherein the session includes one or more previous framesassociated with previous user interactions; creating a working frameincluding one or more intents and one or more concepts expressed in theprevious user interactions and included in the one or more previousframes associated with previous user interactions; sending a request toa natural language processing extraction service, wherein the requestincludes the current interaction; receiving a response from the naturallanguage processing extraction service including a determined concept;evaluating the determined concept and the one or more concepts includedin the previous frames to determine a canonical concept for the workingframe; sending a request to a destination data service, the requestincluding the canonical concept determined for the working frame, thedestination data service determining a destination associated with thecanonical concept by: determining a score for the canonical concept foreach destination in the set of destinations, wherein determining a scorefor the canonical concept for the destination comprises accessing thedestination concept vector associated with the destination to determinethe score for the canonical concept for the destination and selectingthe destination based on the score for the canonical concept for eachdestination in the set of destinations; determining content related tothe canonical concept and the destination, including an image, wheredetermining an image comprises: accessing the set of images associatedwith the destination; determining a score associated with thecanconincal concept for each of the set of images based on the imageconcept vector associated with each of the set of images; and selectingthe image from the set of images based on the score associated with thecanonical concept for each of the set of images; receiving the contentrelated to the canonical concept and the destination from thedestination data service; applying a set of follow-up rules to thecurrent interaction and the previous interaction to determine afollow-up interaction; forming a response to the widget including thecontent related to the canonical concept, destination and the follow-upinteraction; and returning the response to the widget at the userdevice, to be rendered at the user device in the interface displayed atthe user device by presenting the follow-up interaction and the contentin the interface such that the content presented in the interface issynchronized with the canonical concept.
 2. The system of claim 1,wherein the image includes a first image and a second image, wherein thesecond image image is selected from the set of images after theselection of the first image and the second image comprises determininga second score associated with the canconincal concept for each of theset of images based on the image concept vector associated with each ofthe set of images and applying a scoring penalty determined based on asimilarity between the first image.
 3. The system of claim 1, whereinthe image concept vector for the image was determined by: determiningone or more concepts associated with the image; determining a score forthe one or more concepts associated with the image and applying thescore for the one or more concepts associated with the image to aconcept net relationship matrix.
 4. The system of claim 3, wherein theconcept net relationship matrix includes a score between each of a setof concepts utilized by the artificial intelligence system.
 5. Thesystem of claim 4, wherein the score between each of the set of conceptsin the concept net relationship matrix is determined based on a conceptnetwork representing the set of concepts and a set of relationshipsbetween the set of concepts.
 6. The system of claim 5, wherein theconcept network is a trie and the score between a first concept and asecond concept in the concept net relationship matrix is determinedbased on a traversal distance between a first node representing thefirst concept in the concept network and a second node representing thefirst concept in the concept network.
 7. A non-transitory computerreadable medium, comprising instructions for implementing an artificialintelligence system, the instructions executable for: receiving arequest from a widget at a user device, the request including a currentinteraction entered by a user in an interface displayed at the userdevice; retrieving a session associated with the user, wherein thesession includes one or more previous frames associated with previoususer interactions; creating a working frame including one or moreintents and one or more concepts expressed in the previous userinteractions and included in the one or more previous frames associatedwith previous user interactions; sending a request to a natural languageprocessing extraction service, wherein the request includes the currentinteraction; receiving a response from the natural language processingextraction service including a determined concept; evaluating thedetermined concept and the one or more concepts included in the previousframes to determine a canonical concept for the working frame; sending arequest to a destination data service, the request including thecanonical concept determined for the working frame, the destination dataservice determining a destination associated with the canonical conceptby: determining a score for the canonical concept for each destinationin the set of destinations, wherein determining a score for thecanonical concept for the destination comprises accessing a destinationconcept vector associated with the destination to determine the scorefor the canonical concept for the destination and selecting thedestination based on the score for the canonical concept for eachdestination in the set of destinations; determining content related tothe canonical concept and the destination, including an image, wheredetermining an image comprises: accessing a set of images associatedwith the destination; determining a score associated with the canonicalconcept for each of the set of images based on an image concept vectorassociated with each of the set of images; and selecting the image fromthe set of images based on the score associated with the canonicalconcept for each of the set of images; receiving the content related tothe canonical concept and the destination from the destination dataservice; applying a set of follow-up rules to the current interactionand the previous interaction to determine a follow-up interaction;forming a response to the widget including the content related to thecanonical concept, destination and the follow-up interaction; andreturning the response to the widget at the user device, to be renderedat the user device in the interface displayed at the user device bypresenting the follow-up interaction and the content in the interfacesuch that the content presented in the interface is synchronized withthe canonical concept.
 8. The non-transitory computer readable medium ofclaim 7, wherein the image includes a first image and a second image,wherein the second image is selected from the set of images after theselection of the first image and the second image comprises determininga second score associated with the canonical concept for each of the setof images based on the image concept vector associated with each of theset of images and applying a scoring penalty determined based on asimilarity between the first image.
 9. The non-transitory computerreadable medium of claim 7, wherein the image concept vector for theimage was determined by: determining one or more concepts associatedwith the image; determining a score for the one or more conceptsassociated with the image and applying the score for the one or moreconcepts associated with the image to concept net relationship matrix.10. The non-transitory computer readable medium of claim 9, wherein theconcept net relationship matrix includes a score between each of a setof concepts utilized by the artificial intelligence system.
 11. Thenon-transitory computer readable medium of claim 10, wherein the scorebetween each of the set of concepts in the concept net relationshipmatrix is determined based on a concept network representing the set ofconcepts and a set of relationships between the set of concepts.
 12. Thenon-transitory computer readable medium of claim 11, wherein the conceptnetwork is a trie and the score between a first concept and a secondconcept in the concept net relationship matrix is determined based on atraversal distance between a first node representing the first conceptin the concept network and a second node representing the first conceptin the concept network.
 13. A method for an artificial intelligencesystem, comprising: receiving a request from a widget at a user device,the request including a current interaction entered by a user in aninterface displayed at the user device; retrieving a session associatedwith the user, wherein the session includes one or more previous framesassociated with previous user interactions; creating a working frameincluding one or more intents and one or more concepts expressed in theprevious user interactions and included in the one or more previousframes associated with previous user interactions; sending a request toa natural language processing extraction service, wherein the requestincludes the current interaction; receiving a response from the naturallanguage processing extraction service including a determined concept;evaluating the determined concept and the one or more concepts includedin the previous frames to determine a canonical concept for the workingframe; sending a request to a destination data service, the requestincluding the canonical concept determined for the working frame, thedestination data service determining a destination associated with thecanonical concept by: determining a score for the canonical concept foreach destination in the set of destinations, wherein determining a scorefor the canonical concept for the destination comprises accessing adestination concept vector associated with the destination to determinethe score for the canonical concept for the destination and selectingthe destination based on the score for the canonical concept for eachdestination in the set of destinations; determining content related tothe canonical concept and the destination, including an image, wheredetermining an image comprises: accessing a set of images associatedwith the destination; determining a score associated with the canonicalconcept for each of the set of images based on an image concept vectorassociated with each of the set of images; and selecting the image fromthe set of images based on the score associated with the canonicalconcept for each of the set of images; receiving the content related tothe canonical concept and the destination from the destination dataservice; applying a set of follow-up rules to the current interactionand the previous interaction to determine a follow-up interaction;forming a response to the widget including the content related to thecanonical concept, destination and the follow-up interaction; andreturning the response to the widget at the user device, to be renderedat the user device in the interface displayed at the user device bypresenting the follow-up interaction and the content in the interfacesuch that the content presented in the interface is synchronized withthe canonical concept.
 14. The method of claim 13, wherein the imageincludes a first image and a second image, wherein the second image isselected from the set of images after the selection of the first imageand the second image comprises determining a second score associatedwith the canonical concept for each of the set of images based on theimage concept vector associated with each of the set of images andapplying a scoring penalty determined based on a similarity between thefirst image.
 15. The method of claim 13, wherein the image conceptvector for the image was determined by: determining one or more conceptsassociated with the image; determining a score for the one or moreconcepts associated with the image and applying the score for the one ormore concepts associated with the image to concept net relationshipmatrix.
 16. The method of claim 15, wherein the concept net relationshipmatrix includes a score between each of a set of concepts utilized bythe artificial intelligence system.
 17. The method of claim 16, whereinthe score between each of the set of concepts in the concept netrelationship matrix is determined based on a concept networkrepresenting the set of concepts and a set of relationships between theset of concepts.
 18. The method of claim 17, wherein the concept networkis a trie and the score between a first concept and a second concept inthe concept net relationship matrix is determined based on a traversaldistance between a first node representing the first concept in theconcept network and a second node representing the first concept in theconcept network.